2024-03-24T23:44:58.275 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T23:44:58.276 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} --archive /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620593 --verbose --name yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620593/orig.config.yaml 2024-03-24T23:44:58.469 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T23:44:58.551 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620593 branch: wip-yuri10-testing-2024-03-24-1159 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} email: yweinste@redhat.com first_in_suite: false job_id: '7620593' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri10-testing-2024-03-24-1159 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f workunit: branch: wip-yuri10-testing-2024-03-24-1159 sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 108 sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri10-testing-2024-03-24-1159 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f targets: smithi080.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMr7pn2Kl7j8q7QsNf7SVQKcOEdbGDMAKNatw3P8K0C0uMAJ+Eu1kK8JyWhd/6H4iNZ1wwAJMJN0XMd48o7M2zM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-24_22:19:24 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T23:44:58.552 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa; will attempt to use it 2024-03-24T23:44:58.552 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks 2024-03-24T23:44:58.552 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T23:44:58.567 INFO:teuthology.task.internal:Checking packages... 2024-03-24T23:44:58.594 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '9c429fb6abfb5deed62697d56d8997d0f0d6d83f' 2024-03-24T23:44:58.594 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T23:44:58.595 INFO:teuthology.packaging:ref: None 2024-03-24T23:44:58.595 INFO:teuthology.packaging:tag: None 2024-03-24T23:44:58.595 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-03-24-1159 2024-03-24T23:44:58.595 INFO:teuthology.packaging:sha1: 9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-24T23:44:58.595 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri10-testing-2024-03-24-1159 2024-03-24T23:44:58.741 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2398-g9c429fb6-1jammy 2024-03-24T23:44:58.743 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T23:44:58.791 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T23:44:58.791 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T23:44:58.831 INFO:teuthology.task.internal:Saving configuration 2024-03-24T23:44:58.852 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T23:44:58.867 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T23:44:58.893 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620593', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 23:39:32.597008', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDRDCZlaFCA5QEoKk+QqDytkjqMvIqZqleWPtueT8LgM'} 2024-03-24T23:44:58.893 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T23:44:59.031 INFO:teuthology.task.internal:roles: ubuntu@smithi080.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-24T23:44:59.031 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T23:44:59.129 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feef4b6d1f0>, signals=[15]) 2024-03-24T23:44:59.130 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T23:44:59.140 INFO:teuthology.task.internal:Opening connections... 2024-03-24T23:44:59.140 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi080.front.sepia.ceph.com 2024-03-24T23:44:59.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T23:44:59.228 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T23:44:59.237 DEBUG:teuthology.orchestra.run.smithi080:> uname -m 2024-03-24T23:44:59.343 INFO:teuthology.orchestra.run.smithi080.stdout:x86_64 2024-03-24T23:44:59.344 DEBUG:teuthology.orchestra.run.smithi080:> cat /etc/os-release 2024-03-24T23:44:59.392 INFO:teuthology.orchestra.run.smithi080.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-24T23:44:59.392 INFO:teuthology.orchestra.run.smithi080.stdout:NAME="Ubuntu" 2024-03-24T23:44:59.392 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_ID="22.04" 2024-03-24T23:44:59.392 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-24T23:44:59.393 INFO:teuthology.orchestra.run.smithi080.stdout:VERSION_CODENAME=jammy 2024-03-24T23:44:59.393 INFO:teuthology.orchestra.run.smithi080.stdout:ID=ubuntu 2024-03-24T23:44:59.393 INFO:teuthology.orchestra.run.smithi080.stdout:ID_LIKE=debian 2024-03-24T23:44:59.393 INFO:teuthology.orchestra.run.smithi080.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-24T23:44:59.393 INFO:teuthology.orchestra.run.smithi080.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-24T23:44:59.393 INFO:teuthology.orchestra.run.smithi080.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-24T23:44:59.393 INFO:teuthology.orchestra.run.smithi080.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-24T23:44:59.393 INFO:teuthology.orchestra.run.smithi080.stdout:UBUNTU_CODENAME=jammy 2024-03-24T23:44:59.394 INFO:teuthology.lock.ops:Updating smithi080.front.sepia.ceph.com on lock server 2024-03-24T23:44:59.415 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T23:44:59.427 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T23:44:59.436 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T23:44:59.436 DEBUG:teuthology.orchestra.run.smithi080:> test '!' -e /home/ubuntu/cephtest 2024-03-24T23:44:59.440 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T23:44:59.448 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T23:44:59.449 DEBUG:teuthology.orchestra.run.smithi080:> test -z $(ls -A /var/lib/ceph) 2024-03-24T23:44:59.568 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T23:44:59.600 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T23:44:59.614 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T23:44:59.614 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T23:44:59.615 DEBUG:teuthology.orchestra.run.smithi080:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T23:44:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T23:44:59.620 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-03-24T23:44:59.665 INFO:teuthology.orchestra.run.smithi080.stdout:5.15.0-56-generic 2024-03-24T23:44:59.666 INFO:teuthology.task.kernel:Running kernel on smithi080: 5.15.0-56-generic 2024-03-24T23:44:59.666 DEBUG:teuthology.orchestra.run.smithi080:> sudo apt-get clean 2024-03-24T23:44:59.787 DEBUG:teuthology.orchestra.run.smithi080:> sudo apt-get update 2024-03-24T23:44:59.933 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-24T23:44:59.942 INFO:teuthology.orchestra.run.smithi080.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-24T23:44:59.942 INFO:teuthology.orchestra.run.smithi080.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-24T23:45:00.024 INFO:teuthology.orchestra.run.smithi080.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-24T23:45:00.178 INFO:teuthology.orchestra.run.smithi080.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,286 kB] 2024-03-24T23:45:00.252 INFO:teuthology.orchestra.run.smithi080.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,502 kB] 2024-03-24T23:45:00.321 INFO:teuthology.orchestra.run.smithi080.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [428 kB] 2024-03-24T23:45:00.369 INFO:teuthology.orchestra.run.smithi080.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [594 kB] 2024-03-24T23:45:00.376 INFO:teuthology.orchestra.run.smithi080.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [289 kB] 2024-03-24T23:45:00.378 INFO:teuthology.orchestra.run.smithi080.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-24T23:45:00.379 INFO:teuthology.orchestra.run.smithi080.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,619 kB] 2024-03-24T23:45:00.400 INFO:teuthology.orchestra.run.smithi080.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-24T23:45:00.400 INFO:teuthology.orchestra.run.smithi080.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [271 kB] 2024-03-24T23:45:00.404 INFO:teuthology.orchestra.run.smithi080.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-24T23:45:00.405 INFO:teuthology.orchestra.run.smithi080.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [697 kB] 2024-03-24T23:45:00.412 INFO:teuthology.orchestra.run.smithi080.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,059 kB] 2024-03-24T23:45:00.422 INFO:teuthology.orchestra.run.smithi080.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [240 kB] 2024-03-24T23:45:00.424 INFO:teuthology.orchestra.run.smithi080.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-24T23:45:00.449 INFO:teuthology.orchestra.run.smithi080.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-24T23:45:00.449 INFO:teuthology.orchestra.run.smithi080.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-24T23:45:00.450 INFO:teuthology.orchestra.run.smithi080.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-24T23:45:00.450 INFO:teuthology.orchestra.run.smithi080.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-24T23:45:00.450 INFO:teuthology.orchestra.run.smithi080.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-24T23:45:00.451 INFO:teuthology.orchestra.run.smithi080.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-24T23:45:00.451 INFO:teuthology.orchestra.run.smithi080.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-24T23:45:00.463 INFO:teuthology.orchestra.run.smithi080.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-24T23:45:00.472 INFO:teuthology.orchestra.run.smithi080.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [229 kB] 2024-03-24T23:45:00.483 INFO:teuthology.orchestra.run.smithi080.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-24T23:45:00.484 INFO:teuthology.orchestra.run.smithi080.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-24T23:45:00.485 INFO:teuthology.orchestra.run.smithi080.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-24T23:45:00.485 INFO:teuthology.orchestra.run.smithi080.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-24T23:45:00.503 INFO:teuthology.orchestra.run.smithi080.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-24T23:45:00.524 INFO:teuthology.orchestra.run.smithi080.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-24T23:45:00.546 INFO:teuthology.orchestra.run.smithi080.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,591 kB] 2024-03-24T23:45:00.619 INFO:teuthology.orchestra.run.smithi080.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [267 kB] 2024-03-24T23:45:00.645 INFO:teuthology.orchestra.run.smithi080.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-24T23:45:00.666 INFO:teuthology.orchestra.run.smithi080.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-24T23:45:00.701 INFO:teuthology.orchestra.run.smithi080.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [851 kB] 2024-03-24T23:45:00.736 INFO:teuthology.orchestra.run.smithi080.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-24T23:45:00.760 INFO:teuthology.orchestra.run.smithi080.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-24T23:45:00.782 INFO:teuthology.orchestra.run.smithi080.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-24T23:45:00.803 INFO:teuthology.orchestra.run.smithi080.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-24T23:45:00.825 INFO:teuthology.orchestra.run.smithi080.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-24T23:45:00.846 INFO:teuthology.orchestra.run.smithi080.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-24T23:45:05.884 INFO:teuthology.orchestra.run.smithi080.stdout:Fetched 12.5 MB in 2s (6,932 kB/s) 2024-03-24T23:45:07.273 INFO:teuthology.orchestra.run.smithi080.stdout:Reading package lists... 2024-03-24T23:45:07.290 DEBUG:teuthology.orchestra.run.smithi080:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-24T23:45:07.348 INFO:teuthology.orchestra.run.smithi080.stdout:Reading package lists... 2024-03-24T23:45:07.529 INFO:teuthology.orchestra.run.smithi080.stdout:Building dependency tree... 2024-03-24T23:45:07.529 INFO:teuthology.orchestra.run.smithi080.stdout:Reading state information... 2024-03-24T23:45:07.683 INFO:teuthology.orchestra.run.smithi080.stdout:The following packages were automatically installed and are no longer required: 2024-03-24T23:45:07.683 INFO:teuthology.orchestra.run.smithi080.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-24T23:45:07.683 INFO:teuthology.orchestra.run.smithi080.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-24T23:45:07.683 INFO:teuthology.orchestra.run.smithi080.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-24T23:45:07.683 INFO:teuthology.orchestra.run.smithi080.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-24T23:45:07.684 INFO:teuthology.orchestra.run.smithi080.stdout: python2.7-minimal 2024-03-24T23:45:07.684 INFO:teuthology.orchestra.run.smithi080.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-24T23:45:07.685 INFO:teuthology.orchestra.run.smithi080.stdout:The following additional packages will be installed: 2024-03-24T23:45:07.685 INFO:teuthology.orchestra.run.smithi080.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-24T23:45:07.685 INFO:teuthology.orchestra.run.smithi080.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-24T23:45:07.685 INFO:teuthology.orchestra.run.smithi080.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-24T23:45:07.686 INFO:teuthology.orchestra.run.smithi080.stdout:Suggested packages: 2024-03-24T23:45:07.686 INFO:teuthology.orchestra.run.smithi080.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-24T23:45:07.686 INFO:teuthology.orchestra.run.smithi080.stdout:Recommended packages: 2024-03-24T23:45:07.687 INFO:teuthology.orchestra.run.smithi080.stdout: thermald 2024-03-24T23:45:07.734 INFO:teuthology.orchestra.run.smithi080.stdout:The following NEW packages will be installed: 2024-03-24T23:45:07.734 INFO:teuthology.orchestra.run.smithi080.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-24T23:45:07.734 INFO:teuthology.orchestra.run.smithi080.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-24T23:45:07.734 INFO:teuthology.orchestra.run.smithi080.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-24T23:45:07.735 INFO:teuthology.orchestra.run.smithi080.stdout:The following packages will be upgraded: 2024-03-24T23:45:07.736 INFO:teuthology.orchestra.run.smithi080.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-24T23:45:07.792 INFO:teuthology.orchestra.run.smithi080.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-03-24T23:45:07.793 INFO:teuthology.orchestra.run.smithi080.stdout:Need to get 113 MB of archives. 2024-03-24T23:45:07.793 INFO:teuthology.orchestra.run.smithi080.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-24T23:45:07.793 INFO:teuthology.orchestra.run.smithi080.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-24T23:45:08.256 INFO:teuthology.orchestra.run.smithi080.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-24T23:45:08.377 INFO:teuthology.orchestra.run.smithi080.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-24T23:45:09.313 INFO:teuthology.orchestra.run.smithi080.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-24T23:45:09.313 INFO:teuthology.orchestra.run.smithi080.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-24T23:45:09.313 INFO:teuthology.orchestra.run.smithi080.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-24T23:45:09.450 INFO:teuthology.orchestra.run.smithi080.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-24T23:45:09.486 INFO:teuthology.orchestra.run.smithi080.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-24T23:45:09.870 INFO:teuthology.orchestra.run.smithi080.stdout:Fetched 113 MB in 2s (64.9 MB/s) 2024-03-24T23:45:10.050 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-24T23:45:12.198 INFO:teuthology.orchestra.run.smithi080.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-24T23:45:12.203 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-24T23:45:12.472 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-24T23:45:14.607 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-24T23:45:14.628 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-24T23:45:14.735 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-24T23:45:15.176 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-24T23:45:15.197 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-24T23:45:15.234 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-24T23:45:21.875 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-24T23:45:22.002 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-24T23:45:22.335 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-24T23:45:22.481 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-24T23:45:22.742 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-24T23:45:22.764 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-24T23:45:22.800 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-24T23:45:28.151 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-24T23:45:28.188 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-24T23:45:28.226 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-24T23:45:30.945 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-24T23:45:31.070 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-24T23:45:31.404 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-24T23:45:31.508 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-24T23:45:31.618 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-24T23:45:31.746 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-24T23:45:33.054 INFO:teuthology.orchestra.run.smithi080.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-24T23:45:33.055 INFO:teuthology.orchestra.run.smithi080.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-24T23:45:33.055 INFO:teuthology.orchestra.run.smithi080.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-24T23:45:33.055 INFO:teuthology.orchestra.run.smithi080.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-24T23:45:33.233 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-24T23:45:34.607 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-24T23:45:36.193 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-24T23:45:36.318 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-24T23:45:36.445 INFO:teuthology.orchestra.run.smithi080.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-24T23:45:36.512 INFO:teuthology.orchestra.run.smithi080.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-24T23:45:36.512 INFO:teuthology.orchestra.run.smithi080.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-24T23:45:51.353 INFO:teuthology.orchestra.run.smithi080.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-24T23:45:51.354 INFO:teuthology.orchestra.run.smithi080.stdout:Sourcing file `/etc/default/grub' 2024-03-24T23:45:51.370 INFO:teuthology.orchestra.run.smithi080.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-24T23:45:51.391 INFO:teuthology.orchestra.run.smithi080.stdout:Generating grub configuration file ... 2024-03-24T23:45:51.690 INFO:teuthology.orchestra.run.smithi080.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-24T23:45:51.702 INFO:teuthology.orchestra.run.smithi080.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-24T23:45:52.021 INFO:teuthology.orchestra.run.smithi080.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-24T23:45:52.027 INFO:teuthology.orchestra.run.smithi080.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-24T23:45:52.101 INFO:teuthology.orchestra.run.smithi080.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-24T23:45:52.106 INFO:teuthology.orchestra.run.smithi080.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-24T23:45:52.273 INFO:teuthology.orchestra.run.smithi080.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-24T23:45:52.273 INFO:teuthology.orchestra.run.smithi080.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-24T23:45:52.273 INFO:teuthology.orchestra.run.smithi080.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-24T23:45:52.315 INFO:teuthology.orchestra.run.smithi080.stdout:done 2024-03-24T23:45:52.752 DEBUG:teuthology.orchestra.run.smithi080:> dpkg -s linux-image-generic 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Package: linux-image-generic 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Status: install ok installed 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Priority: optional 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Section: kernel 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Installed-Size: 21 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Maintainer: Ubuntu Kernel Team 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Architecture: amd64 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Source: linux-meta 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Version: 5.15.0.101.98 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-24T23:45:52.771 INFO:teuthology.orchestra.run.smithi080.stdout:Recommends: thermald 2024-03-24T23:45:52.772 INFO:teuthology.orchestra.run.smithi080.stdout:Description: Generic Linux kernel image 2024-03-24T23:45:52.772 INFO:teuthology.orchestra.run.smithi080.stdout: This package will always depend on the latest generic kernel image 2024-03-24T23:45:52.772 INFO:teuthology.orchestra.run.smithi080.stdout: available. 2024-03-24T23:45:52.772 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-24T23:45:52.772 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T23:45:52.772 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T23:45:52.772 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T23:45:52.772 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi080.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T23:45:52.773 DEBUG:teuthology.orchestra.run.smithi080:> sudo apt-get clean 2024-03-24T23:45:52.928 DEBUG:teuthology.orchestra.run.smithi080:> sudo apt-get update 2024-03-24T23:45:53.045 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-24T23:45:53.244 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-24T23:45:53.325 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-24T23:45:53.406 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-24T23:45:54.538 INFO:teuthology.orchestra.run.smithi080.stdout:Reading package lists... 2024-03-24T23:45:54.555 DEBUG:teuthology.orchestra.run.smithi080:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-24T23:45:54.616 INFO:teuthology.orchestra.run.smithi080.stdout:Reading package lists... 2024-03-24T23:45:54.813 INFO:teuthology.orchestra.run.smithi080.stdout:Building dependency tree... 2024-03-24T23:45:54.814 INFO:teuthology.orchestra.run.smithi080.stdout:Reading state information... 2024-03-24T23:45:54.982 INFO:teuthology.orchestra.run.smithi080.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-24T23:45:54.982 INFO:teuthology.orchestra.run.smithi080.stdout:The following packages were automatically installed and are no longer required: 2024-03-24T23:45:54.982 INFO:teuthology.orchestra.run.smithi080.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-24T23:45:54.983 INFO:teuthology.orchestra.run.smithi080.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-24T23:45:54.983 INFO:teuthology.orchestra.run.smithi080.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-24T23:45:54.983 INFO:teuthology.orchestra.run.smithi080.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-24T23:45:54.983 INFO:teuthology.orchestra.run.smithi080.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-24T23:45:54.983 INFO:teuthology.orchestra.run.smithi080.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-24T23:45:55.021 INFO:teuthology.orchestra.run.smithi080.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-03-24T23:45:55.023 DEBUG:teuthology.orchestra.run.smithi080:> dpkg -s linux-image-generic 2024-03-24T23:45:55.038 INFO:teuthology.orchestra.run.smithi080.stdout:Package: linux-image-generic 2024-03-24T23:45:55.038 INFO:teuthology.orchestra.run.smithi080.stdout:Status: install ok installed 2024-03-24T23:45:55.038 INFO:teuthology.orchestra.run.smithi080.stdout:Priority: optional 2024-03-24T23:45:55.038 INFO:teuthology.orchestra.run.smithi080.stdout:Section: kernel 2024-03-24T23:45:55.038 INFO:teuthology.orchestra.run.smithi080.stdout:Installed-Size: 21 2024-03-24T23:45:55.038 INFO:teuthology.orchestra.run.smithi080.stdout:Maintainer: Ubuntu Kernel Team 2024-03-24T23:45:55.038 INFO:teuthology.orchestra.run.smithi080.stdout:Architecture: amd64 2024-03-24T23:45:55.038 INFO:teuthology.orchestra.run.smithi080.stdout:Source: linux-meta 2024-03-24T23:45:55.038 INFO:teuthology.orchestra.run.smithi080.stdout:Version: 5.15.0.101.98 2024-03-24T23:45:55.039 INFO:teuthology.orchestra.run.smithi080.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-24T23:45:55.039 INFO:teuthology.orchestra.run.smithi080.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-24T23:45:55.039 INFO:teuthology.orchestra.run.smithi080.stdout:Recommends: thermald 2024-03-24T23:45:55.039 INFO:teuthology.orchestra.run.smithi080.stdout:Description: Generic Linux kernel image 2024-03-24T23:45:55.039 INFO:teuthology.orchestra.run.smithi080.stdout: This package will always depend on the latest generic kernel image 2024-03-24T23:45:55.039 INFO:teuthology.orchestra.run.smithi080.stdout: available. 2024-03-24T23:45:55.039 DEBUG:teuthology.orchestra.run.smithi080:> mktemp 2024-03-24T23:45:55.084 INFO:teuthology.orchestra.run.smithi080.stdout:/tmp/tmp.5GJLiFJ9tG 2024-03-24T23:45:55.084 DEBUG:teuthology.orchestra.run.smithi080:> sudo cp /boot/grub/grub.cfg /tmp/tmp.5GJLiFJ9tG 2024-03-24T23:45:55.139 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 0666 /tmp/tmp.5GJLiFJ9tG 2024-03-24T23:45:55.256 DEBUG:teuthology.orchestra.remote:smithi080:/tmp/tmp.5GJLiFJ9tG is 10KB 2024-03-24T23:45:55.269 DEBUG:teuthology.orchestra.run.smithi080:> rm -fr /tmp/tmp.5GJLiFJ9tG 2024-03-24T23:45:55.274 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-24T23:45:55.327 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-24T23:45:55.327 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-24T23:45:55.327 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-24T23:45:55.390 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-24T23:45:55.390 DEBUG:teuthology.orchestra.run.smithi080:> sudo update-grub 2024-03-24T23:45:56.626 INFO:teuthology.orchestra.run.smithi080.stderr:Sourcing file `/etc/default/grub' 2024-03-24T23:45:56.627 INFO:teuthology.orchestra.run.smithi080.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-24T23:45:56.629 INFO:teuthology.orchestra.run.smithi080.stderr:Generating grub configuration file ... 2024-03-24T23:45:56.788 INFO:teuthology.orchestra.run.smithi080.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-24T23:45:56.794 INFO:teuthology.orchestra.run.smithi080.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-24T23:45:57.022 INFO:teuthology.orchestra.run.smithi080.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-24T23:45:57.028 INFO:teuthology.orchestra.run.smithi080.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-24T23:45:57.070 INFO:teuthology.orchestra.run.smithi080.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-24T23:45:57.075 INFO:teuthology.orchestra.run.smithi080.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-24T23:45:57.188 INFO:teuthology.orchestra.run.smithi080.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-24T23:45:57.188 INFO:teuthology.orchestra.run.smithi080.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-24T23:45:57.188 INFO:teuthology.orchestra.run.smithi080.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-24T23:45:57.217 INFO:teuthology.orchestra.run.smithi080.stderr:done 2024-03-24T23:45:57.225 DEBUG:teuthology.orchestra.run.smithi080:> sudo shutdown -r now 2024-03-24T23:46:27.258 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T23:46:27.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-24T23:46:27.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T23:47:27.323 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T23:47:36.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-24T23:47:36.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T23:47:36.742 ERROR:teuthology.orchestra.connection:Error authenticating with smithi080.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 794, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1664, in auth_publickey raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2024-03-24T23:47:40.826 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-03-24T23:47:41.504 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-24T23:47:41.504 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-101-generic"... 2024-03-24T23:47:41.504 DEBUG:teuthology.orchestra.run.smithi080:> uname -r 2024-03-24T23:47:41.550 INFO:teuthology.orchestra.run.smithi080.stdout:5.15.0-101-generic 2024-03-24T23:47:41.551 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-24T23:47:41.551 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T23:47:41.551 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-24T23:47:42.552 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T23:47:42.552 DEBUG:teuthology.orchestra.run.smithi080:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T23:47:42.657 INFO:teuthology.orchestra.run.smithi080.stdout:ttyS1 2024-03-24T23:47:42.680 DEBUG:teuthology.parallel:result is None 2024-03-24T23:47:42.681 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T23:47:42.688 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T23:47:42.689 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T23:47:42.694 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T23:47:42.701 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T23:47:42.708 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T23:47:42.709 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T23:47:42.762 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T23:47:42.769 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T23:47:42.770 DEBUG:teuthology.orchestra.run.smithi080:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T23:47:42.822 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T23:47:42.830 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T23:47:42.832 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T23:47:42.847 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T23:47:42.847 DEBUG:teuthology.orchestra.run.smithi080:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T23:47:42.910 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T23:47:42.924 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T23:47:42.924 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T23:47:42.963 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T23:47:43.011 DEBUG:teuthology.orchestra.run.smithi080:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T23:47:43.059 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-24T23:47:43.059 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T23:47:43.125 DEBUG:teuthology.orchestra.run.smithi080:> sudo service rsyslog restart 2024-03-24T23:47:43.220 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T23:47:43.227 INFO:teuthology.task.internal:Starting timer... 2024-03-24T23:47:43.228 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T23:47:43.305 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T23:47:43.315 DEBUG:teuthology.task.selinux:Excluding smithi080: OS 'ubuntu' does not support SELinux 2024-03-24T23:47:43.315 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T23:47:43.316 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-24T23:47:43.316 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T23:47:43.316 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T23:47:43.327 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T23:47:43.347 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T23:47:43.348 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi080.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T23:54:42.304 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi080.front.sepia.ceph.com')] 2024-03-24T23:54:42.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-24T23:54:42.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi080.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T23:54:42.390 DEBUG:teuthology.orchestra.run.smithi080:> true 2024-03-24T23:54:42.470 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi080.front.sepia.ceph.com' 2024-03-24T23:54:42.470 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T23:54:42.480 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T23:54:42.480 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T23:54:42.481 DEBUG:teuthology.orchestra.run.smithi080:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T23:54:42.538 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-24T23:54:42.539 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: Command line: ntpd -gq 2024-03-24T23:54:42.539 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: ---------------------------------------------------- 2024-03-24T23:54:42.539 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: ntp-4 is maintained by Network Time Foundation, 2024-03-24T23:54:42.539 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-24T23:54:42.539 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: corporation. Support and training for ntp-4 are 2024-03-24T23:54:42.539 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: available at https://www.nwtime.org/support 2024-03-24T23:54:42.539 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: ---------------------------------------------------- 2024-03-24T23:54:42.540 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: proto: precision = 0.060 usec (-24) 2024-03-24T23:54:42.540 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: basedate set to 2022-02-04 2024-03-24T23:54:42.540 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: gps base set to 2022-02-06 (week 2196) 2024-03-24T23:54:42.540 INFO:teuthology.orchestra.run.smithi080.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-24T23:54:42.541 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-24T23:54:42.541 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: restrict ::: KOD does nothing without LIMITED. 2024-03-24T23:54:42.541 INFO:teuthology.orchestra.run.smithi080.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-24T23:54:42.542 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: Listen and drop on 0 v6wildcard [::]:123 2024-03-24T23:54:42.542 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-24T23:54:42.542 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: Listen normally on 2 lo 127.0.0.1:123 2024-03-24T23:54:42.542 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: Listen normally on 3 enp3s0f1 172.21.15.80:123 2024-03-24T23:54:42.542 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: Listen normally on 4 lo [::1]:123 2024-03-24T23:54:42.542 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cba9%5]:123 2024-03-24T23:54:42.542 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:42 ntpd[17191]: Listening on routing socket on fd #22 for interface updates 2024-03-24T23:54:43.539 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:43 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:43.539 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:43 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:43.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:43 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:43.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:43 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:43.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:43 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T23:54:43.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:43 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T23:54:45.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:45 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:45.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:45 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:45.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:45 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:45.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:45 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:45.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:45 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:45.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:45 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:45.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:45 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T23:54:45.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:45 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T23:54:46.539 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:46 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:46.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:46 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:46.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:46 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:46.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:46 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:46.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:46 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T23:54:46.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:46 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T23:54:47.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:47.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:47.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:47.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:47.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:47.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:47.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T23:54:47.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T23:54:47.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:47.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:47.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T23:54:47.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:47 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T23:54:48.539 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:48 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:48.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:48 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:48.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:48 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:48.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:48 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:48.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:48 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T23:54:48.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:48 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T23:54:49.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:49.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:49.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:49.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:49.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:49.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:49.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T23:54:49.540 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T23:54:49.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T23:54:49.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T23:54:49.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T23:54:49.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T23:54:49.541 INFO:teuthology.orchestra.run.smithi080.stdout:24 Mar 23:54:49 ntpd[17191]: ntpd: time slew -0.001397 s 2024-03-24T23:54:49.541 INFO:teuthology.orchestra.run.smithi080.stdout:ntpd: time slew -0.001397s 2024-03-24T23:54:49.541 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-24T23:54:49.542 INFO:teuthology.orchestra.run.smithi080.stderr:24 Mar 23:54:49 ntpd[17191]: can't open /var/log/ntpstats/loopstats.20240324: Permission denied 2024-03-24T23:54:49.658 INFO:teuthology.orchestra.run.smithi080.stdout: remote refid st t when poll reach delay offset jitter 2024-03-24T23:54:49.658 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================== 2024-03-24T23:54:49.658 INFO:teuthology.orchestra.run.smithi080.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T23:54:49.658 INFO:teuthology.orchestra.run.smithi080.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T23:54:49.658 INFO:teuthology.orchestra.run.smithi080.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T23:54:49.659 INFO:teuthology.orchestra.run.smithi080.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T23:54:49.659 INFO:teuthology.run_tasks:Running task install... 2024-03-24T23:54:49.668 DEBUG:teuthology.task.install:project ceph 2024-03-24T23:54:49.669 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9c429fb6abfb5deed62697d56d8997d0f0d6d83f'}} 2024-03-24T23:54:49.669 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '9c429fb6abfb5deed62697d56d8997d0f0d6d83f'} 2024-03-24T23:54:49.669 INFO:teuthology.task.install:Using flavor: default 2024-03-24T23:54:49.675 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T23:54:49.675 INFO:teuthology.task.install:extra packages: [] 2024-03-24T23:54:49.675 DEBUG:teuthology.orchestra.run.smithi080:> sudo apt-key list | grep Ceph 2024-03-24T23:54:49.759 INFO:teuthology.orchestra.run.smithi080.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-24T23:54:49.785 INFO:teuthology.orchestra.run.smithi080.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-24T23:54:49.785 INFO:teuthology.orchestra.run.smithi080.stdout:uid [ unknown] Ceph.com (release key) 2024-03-24T23:54:49.785 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-24T23:54:49.786 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-24T23:54:50.008 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default/ 2024-03-24T23:54:50.008 INFO:teuthology.task.install.deb:Package version is 19.0.0-2398-g9c429fb6-1jammy 2024-03-24T23:54:50.179 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-24T23:54:50.179 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-24T23:54:50.193 DEBUG:teuthology.orchestra.run.smithi080:> sudo apt-get update 2024-03-24T23:54:50.361 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-24T23:54:50.374 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-24T23:54:50.394 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-24T23:54:50.474 INFO:teuthology.orchestra.run.smithi080.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy InRelease 2024-03-24T23:54:50.480 INFO:teuthology.orchestra.run.smithi080.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-24T23:54:50.511 INFO:teuthology.orchestra.run.smithi080.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-24T23:54:50.548 INFO:teuthology.orchestra.run.smithi080.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-24T23:54:50.585 INFO:teuthology.orchestra.run.smithi080.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,413 B] 2024-03-24T23:54:50.629 INFO:teuthology.orchestra.run.smithi080.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-03-24T23:54:50.706 INFO:teuthology.orchestra.run.smithi080.stdout:Fetched 29.2 kB in 0s (86.4 kB/s) 2024-03-24T23:54:51.600 INFO:teuthology.orchestra.run.smithi080.stdout:Reading package lists... 2024-03-24T23:54:51.615 DEBUG:teuthology.orchestra.run.smithi080:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2398-g9c429fb6-1jammy cephadm=19.0.0-2398-g9c429fb6-1jammy ceph-mds=19.0.0-2398-g9c429fb6-1jammy ceph-mgr=19.0.0-2398-g9c429fb6-1jammy ceph-common=19.0.0-2398-g9c429fb6-1jammy ceph-fuse=19.0.0-2398-g9c429fb6-1jammy ceph-test=19.0.0-2398-g9c429fb6-1jammy ceph-volume=19.0.0-2398-g9c429fb6-1jammy radosgw=19.0.0-2398-g9c429fb6-1jammy python3-rados=19.0.0-2398-g9c429fb6-1jammy python3-rgw=19.0.0-2398-g9c429fb6-1jammy python3-cephfs=19.0.0-2398-g9c429fb6-1jammy python3-rbd=19.0.0-2398-g9c429fb6-1jammy libcephfs2=19.0.0-2398-g9c429fb6-1jammy libcephfs-dev=19.0.0-2398-g9c429fb6-1jammy librados2=19.0.0-2398-g9c429fb6-1jammy librbd1=19.0.0-2398-g9c429fb6-1jammy rbd-fuse=19.0.0-2398-g9c429fb6-1jammy 2024-03-24T23:54:51.674 INFO:teuthology.orchestra.run.smithi080.stdout:Reading package lists... 2024-03-24T23:54:51.867 INFO:teuthology.orchestra.run.smithi080.stdout:Building dependency tree... 2024-03-24T23:54:51.867 INFO:teuthology.orchestra.run.smithi080.stdout:Reading state information... 2024-03-24T23:54:52.040 INFO:teuthology.orchestra.run.smithi080.stdout:The following packages were automatically installed and are no longer required: 2024-03-24T23:54:52.040 INFO:teuthology.orchestra.run.smithi080.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-24T23:54:52.040 INFO:teuthology.orchestra.run.smithi080.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-24T23:54:52.040 INFO:teuthology.orchestra.run.smithi080.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-24T23:54:52.040 INFO:teuthology.orchestra.run.smithi080.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-24T23:54:52.041 INFO:teuthology.orchestra.run.smithi080.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-24T23:54:52.041 INFO:teuthology.orchestra.run.smithi080.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-24T23:54:52.041 INFO:teuthology.orchestra.run.smithi080.stdout:The following additional packages will be installed: 2024-03-24T23:54:52.041 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-24T23:54:52.041 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-24T23:54:52.042 INFO:teuthology.orchestra.run.smithi080.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-24T23:54:52.042 INFO:teuthology.orchestra.run.smithi080.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-24T23:54:52.043 INFO:teuthology.orchestra.run.smithi080.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-24T23:54:52.044 INFO:teuthology.orchestra.run.smithi080.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-24T23:54:52.044 INFO:teuthology.orchestra.run.smithi080.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-24T23:54:52.044 INFO:teuthology.orchestra.run.smithi080.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-24T23:54:52.044 INFO:teuthology.orchestra.run.smithi080.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-24T23:54:52.044 INFO:teuthology.orchestra.run.smithi080.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-24T23:54:52.044 INFO:teuthology.orchestra.run.smithi080.stdout: xmlstarlet zip 2024-03-24T23:54:52.044 INFO:teuthology.orchestra.run.smithi080.stdout:Suggested packages: 2024-03-24T23:54:52.044 INFO:teuthology.orchestra.run.smithi080.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-24T23:54:52.044 INFO:teuthology.orchestra.run.smithi080.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-24T23:54:52.045 INFO:teuthology.orchestra.run.smithi080.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-24T23:54:52.045 INFO:teuthology.orchestra.run.smithi080.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-24T23:54:52.045 INFO:teuthology.orchestra.run.smithi080.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-24T23:54:52.045 INFO:teuthology.orchestra.run.smithi080.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-24T23:54:52.045 INFO:teuthology.orchestra.run.smithi080.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-24T23:54:52.045 INFO:teuthology.orchestra.run.smithi080.stdout:Recommended packages: 2024-03-24T23:54:52.045 INFO:teuthology.orchestra.run.smithi080.stdout: btrfs-tools 2024-03-24T23:54:52.142 INFO:teuthology.orchestra.run.smithi080.stdout:The following NEW packages will be installed: 2024-03-24T23:54:52.142 INFO:teuthology.orchestra.run.smithi080.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-24T23:54:52.143 INFO:teuthology.orchestra.run.smithi080.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-24T23:54:52.144 INFO:teuthology.orchestra.run.smithi080.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-24T23:54:52.144 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-24T23:54:52.144 INFO:teuthology.orchestra.run.smithi080.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-24T23:54:52.144 INFO:teuthology.orchestra.run.smithi080.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-24T23:54:52.144 INFO:teuthology.orchestra.run.smithi080.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-24T23:54:52.144 INFO:teuthology.orchestra.run.smithi080.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-24T23:54:52.144 INFO:teuthology.orchestra.run.smithi080.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-24T23:54:52.144 INFO:teuthology.orchestra.run.smithi080.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-24T23:54:52.144 INFO:teuthology.orchestra.run.smithi080.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-24T23:54:52.144 INFO:teuthology.orchestra.run.smithi080.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-24T23:54:52.144 INFO:teuthology.orchestra.run.smithi080.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-24T23:54:52.205 INFO:teuthology.orchestra.run.smithi080.stdout:0 upgraded, 118 newly installed, 0 to remove and 285 not upgraded. 2024-03-24T23:54:52.205 INFO:teuthology.orchestra.run.smithi080.stdout:Need to get 178 MB of archives. 2024-03-24T23:54:52.205 INFO:teuthology.orchestra.run.smithi080.stdout:After this operation, 818 MB of additional disk space will be used. 2024-03-24T23:54:52.205 INFO:teuthology.orchestra.run.smithi080.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-24T23:54:52.307 INFO:teuthology.orchestra.run.smithi080.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-24T23:54:52.311 INFO:teuthology.orchestra.run.smithi080.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-24T23:54:52.330 INFO:teuthology.orchestra.run.smithi080.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-24T23:54:52.360 INFO:teuthology.orchestra.run.smithi080.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2398-g9c429fb6-1jammy [3,555 kB] 2024-03-24T23:54:52.394 INFO:teuthology.orchestra.run.smithi080.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-24T23:54:52.398 INFO:teuthology.orchestra.run.smithi080.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-24T23:54:52.410 INFO:teuthology.orchestra.run.smithi080.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-24T23:54:52.414 INFO:teuthology.orchestra.run.smithi080.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-24T23:54:52.415 INFO:teuthology.orchestra.run.smithi080.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-24T23:54:52.415 INFO:teuthology.orchestra.run.smithi080.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-24T23:54:52.423 INFO:teuthology.orchestra.run.smithi080.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-24T23:54:52.426 INFO:teuthology.orchestra.run.smithi080.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-24T23:54:52.428 INFO:teuthology.orchestra.run.smithi080.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-24T23:54:52.429 INFO:teuthology.orchestra.run.smithi080.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-24T23:54:52.437 INFO:teuthology.orchestra.run.smithi080.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-24T23:54:52.439 INFO:teuthology.orchestra.run.smithi080.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-24T23:54:52.441 INFO:teuthology.orchestra.run.smithi080.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-24T23:54:52.443 INFO:teuthology.orchestra.run.smithi080.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-24T23:54:52.457 INFO:teuthology.orchestra.run.smithi080.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-24T23:54:52.457 INFO:teuthology.orchestra.run.smithi080.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-24T23:54:52.458 INFO:teuthology.orchestra.run.smithi080.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-24T23:54:52.458 INFO:teuthology.orchestra.run.smithi080.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-24T23:54:52.459 INFO:teuthology.orchestra.run.smithi080.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-24T23:54:52.459 INFO:teuthology.orchestra.run.smithi080.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-24T23:54:52.476 INFO:teuthology.orchestra.run.smithi080.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-24T23:54:52.477 INFO:teuthology.orchestra.run.smithi080.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-24T23:54:52.477 INFO:teuthology.orchestra.run.smithi080.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-24T23:54:52.479 INFO:teuthology.orchestra.run.smithi080.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-24T23:54:52.496 INFO:teuthology.orchestra.run.smithi080.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-24T23:54:52.496 INFO:teuthology.orchestra.run.smithi080.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-24T23:54:52.497 INFO:teuthology.orchestra.run.smithi080.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-24T23:54:52.498 INFO:teuthology.orchestra.run.smithi080.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-24T23:54:52.498 INFO:teuthology.orchestra.run.smithi080.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-24T23:54:52.498 INFO:teuthology.orchestra.run.smithi080.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-24T23:54:52.515 INFO:teuthology.orchestra.run.smithi080.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-24T23:54:52.516 INFO:teuthology.orchestra.run.smithi080.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-24T23:54:52.516 INFO:teuthology.orchestra.run.smithi080.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-24T23:54:52.522 INFO:teuthology.orchestra.run.smithi080.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-24T23:54:52.535 INFO:teuthology.orchestra.run.smithi080.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-24T23:54:52.535 INFO:teuthology.orchestra.run.smithi080.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-24T23:54:52.536 INFO:teuthology.orchestra.run.smithi080.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-24T23:54:52.537 INFO:teuthology.orchestra.run.smithi080.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-24T23:54:52.539 INFO:teuthology.orchestra.run.smithi080.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-24T23:54:52.539 INFO:teuthology.orchestra.run.smithi080.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-24T23:54:52.554 INFO:teuthology.orchestra.run.smithi080.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-24T23:54:52.555 INFO:teuthology.orchestra.run.smithi080.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-24T23:54:52.557 INFO:teuthology.orchestra.run.smithi080.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-24T23:54:52.557 INFO:teuthology.orchestra.run.smithi080.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-24T23:54:52.573 INFO:teuthology.orchestra.run.smithi080.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-24T23:54:52.576 INFO:teuthology.orchestra.run.smithi080.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-24T23:54:52.628 INFO:teuthology.orchestra.run.smithi080.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-24T23:54:52.629 INFO:teuthology.orchestra.run.smithi080.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-24T23:54:52.630 INFO:teuthology.orchestra.run.smithi080.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-24T23:54:52.630 INFO:teuthology.orchestra.run.smithi080.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-24T23:54:52.631 INFO:teuthology.orchestra.run.smithi080.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-24T23:54:52.650 INFO:teuthology.orchestra.run.smithi080.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-24T23:54:52.652 INFO:teuthology.orchestra.run.smithi080.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-24T23:54:52.652 INFO:teuthology.orchestra.run.smithi080.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-24T23:54:52.671 INFO:teuthology.orchestra.run.smithi080.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-24T23:54:52.672 INFO:teuthology.orchestra.run.smithi080.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-24T23:54:52.672 INFO:teuthology.orchestra.run.smithi080.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-24T23:54:52.672 INFO:teuthology.orchestra.run.smithi080.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-24T23:54:52.672 INFO:teuthology.orchestra.run.smithi080.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-24T23:54:52.682 INFO:teuthology.orchestra.run.smithi080.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-24T23:54:52.682 INFO:teuthology.orchestra.run.smithi080.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-24T23:54:52.683 INFO:teuthology.orchestra.run.smithi080.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-24T23:54:52.688 INFO:teuthology.orchestra.run.smithi080.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-24T23:54:52.702 INFO:teuthology.orchestra.run.smithi080.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-24T23:54:52.703 INFO:teuthology.orchestra.run.smithi080.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-24T23:54:52.703 INFO:teuthology.orchestra.run.smithi080.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-24T23:54:52.707 INFO:teuthology.orchestra.run.smithi080.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-24T23:54:52.711 INFO:teuthology.orchestra.run.smithi080.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-24T23:54:52.711 INFO:teuthology.orchestra.run.smithi080.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-24T23:54:52.721 INFO:teuthology.orchestra.run.smithi080.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-24T23:54:52.726 INFO:teuthology.orchestra.run.smithi080.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-24T23:54:52.727 INFO:teuthology.orchestra.run.smithi080.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-24T23:54:52.730 INFO:teuthology.orchestra.run.smithi080.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-24T23:54:52.740 INFO:teuthology.orchestra.run.smithi080.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-24T23:54:52.741 INFO:teuthology.orchestra.run.smithi080.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-24T23:54:52.741 INFO:teuthology.orchestra.run.smithi080.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-24T23:54:52.743 INFO:teuthology.orchestra.run.smithi080.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-24T23:54:52.743 INFO:teuthology.orchestra.run.smithi080.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-24T23:54:52.750 INFO:teuthology.orchestra.run.smithi080.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-24T23:54:52.760 INFO:teuthology.orchestra.run.smithi080.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-24T23:54:52.760 INFO:teuthology.orchestra.run.smithi080.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-24T23:54:52.762 INFO:teuthology.orchestra.run.smithi080.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-24T23:54:52.763 INFO:teuthology.orchestra.run.smithi080.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-24T23:54:52.830 INFO:teuthology.orchestra.run.smithi080.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2398-g9c429fb6-1jammy [3,226 kB] 2024-03-24T23:54:53.018 INFO:teuthology.orchestra.run.smithi080.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2398-g9c429fb6-1jammy [711 kB] 2024-03-24T23:54:53.054 INFO:teuthology.orchestra.run.smithi080.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2398-g9c429fb6-1jammy [356 kB] 2024-03-24T23:54:53.084 INFO:teuthology.orchestra.run.smithi080.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2398-g9c429fb6-1jammy [32.8 kB] 2024-03-24T23:54:53.085 INFO:teuthology.orchestra.run.smithi080.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2398-g9c429fb6-1jammy [180 kB] 2024-03-24T23:54:53.089 INFO:teuthology.orchestra.run.smithi080.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2398-g9c429fb6-1jammy [65.7 kB] 2024-03-24T23:54:53.091 INFO:teuthology.orchestra.run.smithi080.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2398-g9c429fb6-1jammy [330 kB] 2024-03-24T23:54:53.119 INFO:teuthology.orchestra.run.smithi080.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2398-g9c429fb6-1jammy [6,660 kB] 2024-03-24T23:54:53.385 INFO:teuthology.orchestra.run.smithi080.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2398-g9c429fb6-1jammy [112 kB] 2024-03-24T23:54:53.390 INFO:teuthology.orchestra.run.smithi080.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2398-g9c429fb6-1jammy [466 kB] 2024-03-24T23:54:53.410 INFO:teuthology.orchestra.run.smithi080.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2398-g9c429fb6-1jammy [25.3 MB] 2024-03-24T23:54:54.325 INFO:teuthology.orchestra.run.smithi080.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2398-g9c429fb6-1jammy [5,085 kB] 2024-03-24T23:54:54.402 INFO:teuthology.orchestra.run.smithi080.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2398-g9c429fb6-1jammy [244 kB] 2024-03-24T23:54:54.408 INFO:teuthology.orchestra.run.smithi080.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2398-g9c429fb6-1jammy [124 kB] 2024-03-24T23:54:54.412 INFO:teuthology.orchestra.run.smithi080.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2398-g9c429fb6-1jammy [1,526 kB] 2024-03-24T23:54:54.464 INFO:teuthology.orchestra.run.smithi080.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2398-g9c429fb6-1jammy [6,256 kB] 2024-03-24T23:54:54.669 INFO:teuthology.orchestra.run.smithi080.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2398-g9c429fb6-1jammy [22.8 MB] 2024-03-24T23:54:55.600 INFO:teuthology.orchestra.run.smithi080.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2398-g9c429fb6-1jammy [14.0 kB] 2024-03-24T23:54:55.601 INFO:teuthology.orchestra.run.smithi080.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2398-g9c429fb6-1jammy [870 kB] 2024-03-24T23:54:55.618 INFO:teuthology.orchestra.run.smithi080.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2398-g9c429fb6-1jammy [2,434 kB] 2024-03-24T23:54:55.660 INFO:teuthology.orchestra.run.smithi080.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2398-g9c429fb6-1jammy [779 kB] 2024-03-24T23:54:55.670 INFO:teuthology.orchestra.run.smithi080.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2398-g9c429fb6-1jammy [146 kB] 2024-03-24T23:54:55.672 INFO:teuthology.orchestra.run.smithi080.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2398-g9c429fb6-1jammy [2,318 kB] 2024-03-24T23:54:55.716 INFO:teuthology.orchestra.run.smithi080.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2398-g9c429fb6-1jammy [8,625 kB] 2024-03-24T23:54:56.000 INFO:teuthology.orchestra.run.smithi080.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2398-g9c429fb6-1jammy [14.2 kB] 2024-03-24T23:54:56.000 INFO:teuthology.orchestra.run.smithi080.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2398-g9c429fb6-1jammy [50.3 MB] 2024-03-24T23:54:57.906 INFO:teuthology.orchestra.run.smithi080.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2398-g9c429fb6-1jammy [121 kB] 2024-03-24T23:54:57.908 INFO:teuthology.orchestra.run.smithi080.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2398-g9c429fb6-1jammy [38.9 kB] 2024-03-24T23:54:57.910 INFO:teuthology.orchestra.run.smithi080.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2398-g9c429fb6-1jammy [13.2 MB] 2024-03-24T23:54:58.248 INFO:teuthology.orchestra.run.smithi080.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-yuri10-testing-2024-03-24-1159/9c429fb6abfb5deed62697d56d8997d0f0d6d83f/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2398-g9c429fb6-1jammy [91.4 kB] 2024-03-24T23:54:58.801 INFO:teuthology.orchestra.run.smithi080.stdout:Fetched 178 MB in 6s (29.3 MB/s) 2024-03-24T23:54:58.965 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-24T23:54:59.065 INFO:teuthology.orchestra.run.smithi080.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-24T23:54:59.070 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-24T23:54:59.155 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-24T23:54:59.500 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-24T23:54:59.526 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-24T23:54:59.567 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-24T23:54:59.903 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-24T23:54:59.930 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-24T23:54:59.978 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-24T23:55:00.374 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-24T23:55:00.400 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-24T23:55:00.447 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T23:55:00.852 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-24T23:55:00.878 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-24T23:55:00.918 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T23:55:01.247 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-24T23:55:01.273 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-24T23:55:01.314 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T23:55:01.650 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-24T23:55:01.676 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-24T23:55:01.716 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-24T23:55:02.062 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package librados2. 2024-03-24T23:55:02.088 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../007-librados2_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:02.128 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking librados2 (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:02.656 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package librbd1. 2024-03-24T23:55:02.682 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../008-librbd1_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:02.722 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking librbd1 (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:03.167 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libcephfs2. 2024-03-24T23:55:03.193 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:03.234 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libcephfs2 (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:03.511 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-rados. 2024-03-24T23:55:03.526 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:03.570 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-rados (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:03.897 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-24T23:55:03.923 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2398-g9c429fb6-1jammy_all.deb ... 2024-03-24T23:55:03.964 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-ceph-argparse (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:04.208 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-cephfs. 2024-03-24T23:55:04.234 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:04.275 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-cephfs (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:04.561 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-24T23:55:04.587 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2398-g9c429fb6-1jammy_all.deb ... 2024-03-24T23:55:04.628 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-ceph-common (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:04.956 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-24T23:55:04.982 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-24T23:55:05.022 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-24T23:55:05.359 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-prettytable. 2024-03-24T23:55:05.385 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-24T23:55:05.425 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-24T23:55:05.662 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-rbd. 2024-03-24T23:55:05.688 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:05.720 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-rbd (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:06.023 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-24T23:55:06.050 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-24T23:55:06.114 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-24T23:55:06.384 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-24T23:55:06.399 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-24T23:55:06.442 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-24T23:55:06.754 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-24T23:55:06.769 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-24T23:55:06.812 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-24T23:55:07.240 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package lua5.1. 2024-03-24T23:55:07.267 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-24T23:55:07.307 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-24T23:55:07.610 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package lua-any. 2024-03-24T23:55:07.625 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-24T23:55:07.668 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-24T23:55:07.913 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package zip. 2024-03-24T23:55:07.939 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-24T23:55:07.979 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking zip (3.0-12build2) ... 2024-03-24T23:55:08.299 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package unzip. 2024-03-24T23:55:08.325 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-24T23:55:08.357 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-24T23:55:08.753 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package luarocks. 2024-03-24T23:55:08.779 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-24T23:55:08.819 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-24T23:55:09.227 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package librgw2. 2024-03-24T23:55:09.254 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../025-librgw2_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:09.294 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking librgw2 (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:09.830 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-rgw. 2024-03-24T23:55:09.845 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:09.880 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-rgw (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:10.200 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-24T23:55:10.226 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-24T23:55:10.266 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-24T23:55:10.586 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libradosstriper1. 2024-03-24T23:55:10.612 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:10.652 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libradosstriper1 (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:11.340 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-common. 2024-03-24T23:55:11.366 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:11.406 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-common (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:12.761 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-base. 2024-03-24T23:55:12.788 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:12.837 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-base (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:13.424 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-24T23:55:13.453 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-24T23:55:13.491 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-24T23:55:13.802 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-cheroot. 2024-03-24T23:55:13.828 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-24T23:55:13.868 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-24T23:55:14.172 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-24T23:55:14.198 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-24T23:55:14.238 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-24T23:55:14.500 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-24T23:55:14.526 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-24T23:55:14.558 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-24T23:55:14.844 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-24T23:55:14.870 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-24T23:55:14.910 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-24T23:55:15.213 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-tempora. 2024-03-24T23:55:15.240 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-24T23:55:15.303 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-24T23:55:15.591 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-portend. 2024-03-24T23:55:15.618 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-24T23:55:15.649 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-24T23:55:15.936 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-24T23:55:15.962 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-24T23:55:16.003 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-24T23:55:16.306 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-24T23:55:16.332 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-24T23:55:16.372 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-24T23:55:16.693 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-natsort. 2024-03-24T23:55:16.719 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-24T23:55:16.759 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-24T23:55:17.079 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-logutils. 2024-03-24T23:55:17.106 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-24T23:55:17.146 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-24T23:55:17.448 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-24T23:55:17.476 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-24T23:55:17.515 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-24T23:55:17.776 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-mako. 2024-03-24T23:55:17.803 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-24T23:55:17.843 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-24T23:55:18.138 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-24T23:55:18.153 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-24T23:55:18.196 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-24T23:55:18.524 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-24T23:55:18.551 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-24T23:55:18.591 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-24T23:55:18.893 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-webob. 2024-03-24T23:55:18.920 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-24T23:55:18.960 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-24T23:55:19.305 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-waitress. 2024-03-24T23:55:19.332 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-24T23:55:19.374 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-24T23:55:19.683 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-tempita. 2024-03-24T23:55:19.709 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-24T23:55:19.750 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-24T23:55:20.052 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-paste. 2024-03-24T23:55:20.079 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-24T23:55:20.119 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-24T23:55:20.447 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-24T23:55:20.463 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-24T23:55:20.506 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-24T23:55:20.784 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-24T23:55:20.810 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-24T23:55:20.842 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-24T23:55:21.145 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-webtest. 2024-03-24T23:55:21.171 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-24T23:55:21.228 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-24T23:55:21.731 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pecan. 2024-03-24T23:55:21.758 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-24T23:55:21.789 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-24T23:55:22.086 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-certifi. 2024-03-24T23:55:22.112 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-24T23:55:22.152 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-24T23:55:22.489 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-idna. 2024-03-24T23:55:22.515 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-24T23:55:22.556 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-24T23:55:22.867 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-urllib3. 2024-03-24T23:55:22.893 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-24T23:55:22.933 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-24T23:55:23.270 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-requests. 2024-03-24T23:55:23.297 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-24T23:55:23.337 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-24T23:55:23.665 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-24T23:55:23.691 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-24T23:55:23.731 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-24T23:55:24.026 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-24T23:55:24.041 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2398-g9c429fb6-1jammy_all.deb ... 2024-03-24T23:55:24.084 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:24.481 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-24T23:55:24.508 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:24.556 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:24.850 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-24T23:55:24.877 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-24T23:55:24.917 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-24T23:55:25.228 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-openssl. 2024-03-24T23:55:25.255 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-24T23:55:25.295 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-24T23:55:26.457 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr. 2024-03-24T23:55:26.484 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:26.523 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:26.944 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mon. 2024-03-24T23:55:26.970 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:27.010 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mon (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:27.656 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-osd. 2024-03-24T23:55:27.682 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:27.722 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-osd (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:28.758 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph. 2024-03-24T23:55:28.774 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../066-ceph_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:28.817 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:29.145 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-fuse. 2024-03-24T23:55:29.171 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:29.211 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-fuse (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:29.581 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mds. 2024-03-24T23:55:29.608 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:29.648 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mds (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:29.984 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package cephadm. 2024-03-24T23:55:30.010 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../069-cephadm_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:30.050 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking cephadm (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:30.370 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-24T23:55:30.397 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-24T23:55:30.437 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-24T23:55:30.773 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-24T23:55:30.800 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-24T23:55:30.839 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-24T23:55:31.870 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-babel. 2024-03-24T23:55:31.897 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-24T23:55:32.061 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-24T23:55:32.431 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-jinja2. 2024-03-24T23:55:32.458 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-24T23:55:32.497 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-24T23:55:32.851 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-24T23:55:32.877 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2398-g9c429fb6-1jammy_all.deb ... 2024-03-24T23:55:32.917 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:33.254 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-24T23:55:33.281 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-24T23:55:33.321 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-24T23:55:33.623 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-routes. 2024-03-24T23:55:33.650 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-24T23:55:33.690 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-24T23:55:34.027 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-24T23:55:34.054 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2398-g9c429fb6-1jammy_all.deb ... 2024-03-24T23:55:34.093 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:35.323 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-24T23:55:35.339 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-24T23:55:35.373 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-24T23:55:35.877 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-joblib. 2024-03-24T23:55:35.904 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-24T23:55:35.943 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-24T23:55:36.322 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-24T23:55:36.349 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-24T23:55:36.388 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-24T23:55:36.691 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-sklearn. 2024-03-24T23:55:36.718 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-24T23:55:36.758 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-24T23:55:37.435 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-24T23:55:37.451 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2398-g9c429fb6-1jammy_all.deb ... 2024-03-24T23:55:37.569 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:38.566 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-cachetools. 2024-03-24T23:55:38.594 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-24T23:55:38.632 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-24T23:55:38.935 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-24T23:55:38.963 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-24T23:55:39.002 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-24T23:55:39.322 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-24T23:55:39.350 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-24T23:55:39.381 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-24T23:55:39.675 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-rsa. 2024-03-24T23:55:39.702 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-24T23:55:39.742 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-24T23:55:40.079 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-google-auth. 2024-03-24T23:55:40.107 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-24T23:55:40.145 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-24T23:55:40.457 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-24T23:55:40.485 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-24T23:55:40.524 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-24T23:55:40.785 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-websocket. 2024-03-24T23:55:40.813 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-24T23:55:40.851 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-24T23:55:41.172 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-24T23:55:41.200 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-24T23:55:41.238 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-24T23:55:41.911 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-24T23:55:41.927 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2398-g9c429fb6-1jammy_all.deb ... 2024-03-24T23:55:41.969 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:42.289 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-24T23:55:42.317 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-24T23:55:42.355 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-24T23:55:42.675 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-24T23:55:42.702 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-24T23:55:42.742 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-24T23:55:43.003 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package jq. 2024-03-24T23:55:43.030 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-24T23:55:43.069 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-24T23:55:43.322 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package socat. 2024-03-24T23:55:43.346 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-24T23:55:43.389 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-24T23:55:43.683 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package xmlstarlet. 2024-03-24T23:55:43.699 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-24T23:55:43.733 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-24T23:55:44.088 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-test. 2024-03-24T23:55:44.116 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:44.155 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-test (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:46.467 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package ceph-volume. 2024-03-24T23:55:46.495 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2398-g9c429fb6-1jammy_all.deb ... 2024-03-24T23:55:46.525 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking ceph-volume (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:46.821 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-24T23:55:46.849 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:46.887 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking libcephfs-dev (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:47.316 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-24T23:55:47.343 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-24T23:55:47.374 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-24T23:55:47.928 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-24T23:55:47.956 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-24T23:55:47.994 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-24T23:55:48.357 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package nvme-cli. 2024-03-24T23:55:48.385 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-24T23:55:48.424 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-24T23:55:48.862 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package pkg-config. 2024-03-24T23:55:48.890 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-24T23:55:48.929 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-24T23:55:49.199 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-24T23:55:49.226 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-24T23:55:49.265 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-24T23:55:49.694 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-24T23:55:49.722 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-24T23:55:49.760 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-24T23:55:50.029 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pastescript. 2024-03-24T23:55:50.058 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-24T23:55:50.088 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-24T23:55:50.383 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pluggy. 2024-03-24T23:55:50.411 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-24T23:55:50.449 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-24T23:55:50.761 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-psutil. 2024-03-24T23:55:50.789 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-24T23:55:50.827 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-24T23:55:51.138 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-py. 2024-03-24T23:55:51.166 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-24T23:55:51.205 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-24T23:55:51.500 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pygments. 2024-03-24T23:55:51.528 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-24T23:55:51.558 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-24T23:55:51.987 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-24T23:55:52.015 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-24T23:55:52.054 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-24T23:55:52.398 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-toml. 2024-03-24T23:55:52.426 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-24T23:55:52.465 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-24T23:55:52.751 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-pytest. 2024-03-24T23:55:52.779 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-24T23:55:52.817 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-24T23:55:53.396 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package python3-simplejson. 2024-03-24T23:55:53.424 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-24T23:55:53.471 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-24T23:55:53.741 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-24T23:55:53.768 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-24T23:55:53.807 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-24T23:55:54.544 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package radosgw. 2024-03-24T23:55:54.572 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../116-radosgw_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:54.611 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking radosgw (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:55.423 INFO:teuthology.orchestra.run.smithi080.stdout:Selecting previously unselected package rbd-fuse. 2024-03-24T23:55:55.451 INFO:teuthology.orchestra.run.smithi080.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2398-g9c429fb6-1jammy_amd64.deb ... 2024-03-24T23:55:55.489 INFO:teuthology.orchestra.run.smithi080.stdout:Unpacking rbd-fuse (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:55.760 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-24T23:55:56.010 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-24T23:55:56.137 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-24T23:55:56.392 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-24T23:55:56.713 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-24T23:55:57.153 INFO:teuthology.orchestra.run.smithi080.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-24T23:55:57.222 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up cephadm (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:55:57.721 INFO:teuthology.orchestra.run.smithi080.stdout:Adding system user cephadm....done 2024-03-24T23:55:57.853 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-24T23:55:58.130 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-24T23:55:58.372 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-24T23:55:58.473 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-24T23:55:58.716 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-24T23:55:58.976 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-24T23:55:59.102 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-24T23:55:59.421 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-24T23:55:59.773 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-24T23:56:00.015 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-24T23:56:00.275 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-24T23:56:00.527 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-24T23:56:00.778 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-24T23:56:01.046 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-24T23:56:01.305 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-ceph-argparse (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:01.606 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-24T23:56:01.840 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-24T23:56:01.941 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-24T23:56:02.067 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-24T23:56:02.335 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-24T23:56:02.755 INFO:teuthology.orchestra.run.smithi080.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-24T23:56:02.791 INFO:teuthology.orchestra.run.smithi080.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-24T23:56:02.855 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-24T23:56:02.965 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-24T23:56:03.316 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-24T23:56:03.425 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-24T23:56:03.693 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-24T23:56:03.952 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-24T23:56:04.271 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up zip (3.0-12build2) ... 2024-03-24T23:56:04.397 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-24T23:56:05.115 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-24T23:56:05.358 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-24T23:56:05.484 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-24T23:56:05.610 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-24T23:56:05.911 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-24T23:56:06.296 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-24T23:56:06.555 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-24T23:56:06.924 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-24T23:56:07.216 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-24T23:56:07.560 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-24T23:56:07.811 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-24T23:56:07.937 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-ceph-common (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:08.239 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-idna (3.3-1) ... 2024-03-24T23:56:08.548 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-24T23:56:09.629 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-24T23:56:09.889 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T23:56:10.014 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-24T23:56:10.257 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-24T23:56:10.375 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-24T23:56:10.501 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-24T23:56:10.819 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-24T23:56:11.079 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-24T23:56:11.321 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T23:56:11.439 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-24T23:56:11.749 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-24T23:56:12.000 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-24T23:56:12.259 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-24T23:56:12.519 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-24T23:56:12.762 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-24T23:56:13.021 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-24T23:56:13.156 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-24T23:56:13.415 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-24T23:56:13.516 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-24T23:56:14.009 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-24T23:56:14.310 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-24T23:56:14.578 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-24T23:56:14.704 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-24T23:56:14.830 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-24T23:56:14.919 INFO:teuthology.orchestra.run.smithi080.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-24T23:56:15.148 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-24T23:56:15.275 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-24T23:56:15.659 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-24T23:56:15.919 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-24T23:56:16.020 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-24T23:56:16.271 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T23:56:16.389 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-24T23:56:16.681 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-24T23:56:17.000 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-24T23:56:17.267 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-24T23:56:17.385 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-24T23:56:17.628 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-24T23:56:17.888 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-24T23:56:18.232 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-24T23:56:18.592 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-24T23:56:18.869 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-24T23:56:19.171 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-24T23:56:19.273 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up librados2 (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:19.374 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:19.500 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-24T23:56:19.836 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libcephfs2 (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:19.962 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libradosstriper1 (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:20.080 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-24T23:56:20.340 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up librbd1 (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:20.466 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr-modules-core (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:20.592 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-fuse (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:20.802 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-24T23:56:20.802 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-24T23:56:21.326 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up libcephfs-dev (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:21.430 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-rados (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:21.529 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up librgw2 (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:21.655 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-24T23:56:22.825 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-rbd (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:22.976 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up rbd-fuse (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:23.102 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-rgw (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:23.229 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up python3-cephfs (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:23.355 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-common (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:23.839 INFO:teuthology.orchestra.run.smithi080.stdout:Adding group ceph....done 2024-03-24T23:56:24.208 INFO:teuthology.orchestra.run.smithi080.stdout:Adding system user ceph....done 2024-03-24T23:56:24.359 INFO:teuthology.orchestra.run.smithi080.stdout:Setting system user ceph properties....done 2024-03-24T23:56:24.373 INFO:teuthology.orchestra.run.smithi080.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-24T23:56:24.513 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-24T23:56:24.834 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-24T23:56:25.331 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-test (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:25.474 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up radosgw (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:25.984 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-24T23:56:25.984 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-24T23:56:26.467 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-base (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:26.817 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-24T23:56:27.321 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mds (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:27.544 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-24T23:56:27.544 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-24T23:56:28.057 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:28.279 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-24T23:56:28.279 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-24T23:56:28.758 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-osd (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:29.024 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-24T23:56:29.024 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-24T23:56:29.543 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:29.669 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:29.820 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mon (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:30.021 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-24T23:56:30.022 INFO:teuthology.orchestra.run.smithi080.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-24T23:56:30.530 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr-cephadm (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:30.656 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:30.782 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-mgr-dashboard (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:30.933 INFO:teuthology.orchestra.run.smithi080.stdout:Setting up ceph-volume (19.0.0-2398-g9c429fb6-1jammy) ... 2024-03-24T23:56:31.287 INFO:teuthology.orchestra.run.smithi080.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-24T23:56:31.395 INFO:teuthology.orchestra.run.smithi080.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-24T23:56:31.540 INFO:teuthology.orchestra.run.smithi080.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-24T23:56:32.675 INFO:teuthology.orchestra.run.smithi080.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-24T23:56:33.342 INFO:teuthology.orchestra.run.smithi080.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-24T23:56:33.348 DEBUG:teuthology.parallel:result is None 2024-03-24T23:56:33.348 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-24T23:56:33.579 DEBUG:teuthology.orchestra.run.smithi080:> dpkg-query -W -f '${Version}' ceph 2024-03-24T23:56:33.598 INFO:teuthology.orchestra.run.smithi080.stdout:19.0.0-2398-g9c429fb6-1jammy 2024-03-24T23:56:33.598 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2398-g9c429fb6-1jammy 2024-03-24T23:56:33.598 INFO:teuthology.task.install:The correct ceph version 19.0.0-2398-g9c429fb6-1jammy is installed. 2024-03-24T23:56:33.600 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T23:56:33.600 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-24T23:56:33.600 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T23:56:33.653 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T23:56:33.653 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-24T23:56:33.653 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T23:56:33.710 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T23:56:33.763 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T23:56:33.764 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-24T23:56:33.764 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T23:56:33.817 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T23:56:33.872 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T23:56:33.873 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-24T23:56:33.873 DEBUG:teuthology.orchestra.run.smithi080:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T23:56:33.929 DEBUG:teuthology.orchestra.run.smithi080:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T23:56:33.984 INFO:teuthology.run_tasks:Running task workunit... 2024-03-24T23:56:34.006 INFO:tasks.workunit:Pulling workunits from ref 9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-24T23:56:34.007 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T23:56:34.007 INFO:tasks.workunit:timeout=3h 2024-03-24T23:56:34.007 INFO:tasks.workunit:cleanup=True 2024-03-24T23:56:34.008 DEBUG:teuthology.orchestra.run.smithi080:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T23:56:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T23:56:34.034 INFO:teuthology.orchestra.run.smithi080.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-24T23:56:34.034 DEBUG:teuthology.orchestra.run.smithi080:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-24T23:56:34.081 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-24T23:56:34.081 DEBUG:teuthology.orchestra.run.smithi080:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-24T23:56:34.134 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 9c429fb6abfb5deed62697d56d8997d0f0d6d83f 2024-03-24T23:56:34.183 INFO:tasks.workunit.client.0.smithi080.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T23:57:21.852 INFO:tasks.workunit.client.0.smithi080.stderr:Note: switching to '9c429fb6abfb5deed62697d56d8997d0f0d6d83f'. 2024-03-24T23:57:21.852 INFO:tasks.workunit.client.0.smithi080.stderr: 2024-03-24T23:57:21.852 INFO:tasks.workunit.client.0.smithi080.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr: 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr: 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr: git switch -c 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr: 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr:Or undo this operation with: 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr: 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr: git switch - 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr: 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T23:57:21.853 INFO:tasks.workunit.client.0.smithi080.stderr: 2024-03-24T23:57:21.854 INFO:tasks.workunit.client.0.smithi080.stderr:HEAD is now at 9c429fb6abf Merge branch 'wip-osd-ec-partial-reads' of https://github.com/rzarzynski/ceph into wip-yuri10-testing-2024-03-24-1159 2024-03-24T23:57:21.857 DEBUG:teuthology.orchestra.run.smithi080:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T23:57:21.905 DEBUG:teuthology.orchestra.run.smithi080:> set -ex 2024-03-24T23:57:21.906 DEBUG:teuthology.orchestra.run.smithi080:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T23:57:21.953 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2024-03-24T23:57:21.953 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2024-03-24T23:57:21.953 DEBUG:teuthology.orchestra.run.smithi080:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9c429fb6abfb5deed62697d56d8997d0f0d6d83f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2024-03-24T23:57:22.004 INFO:tasks.workunit.client.0.smithi080.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-03-24T23:57:22.004 INFO:tasks.workunit.client.0.smithi080.stderr:++ TIMEOUT=300 2024-03-24T23:57:22.004 INFO:tasks.workunit.client.0.smithi080.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-03-24T23:57:22.004 INFO:tasks.workunit.client.0.smithi080.stderr:++ MAX_TIMEOUT=15 2024-03-24T23:57:22.004 INFO:tasks.workunit.client.0.smithi080.stderr:++ PG_NUM=4 2024-03-24T23:57:22.005 INFO:tasks.workunit.client.0.smithi080.stderr:++ TMPDIR=/tmp 2024-03-24T23:57:22.005 INFO:tasks.workunit.client.0.smithi080.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-03-24T23:57:22.005 INFO:tasks.workunit.client.0.smithi080.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-03-24T23:57:22.005 INFO:tasks.workunit.client.0.smithi080.stderr:++ type xmlstarlet 2024-03-24T23:57:22.005 INFO:tasks.workunit.client.0.smithi080.stderr:++ XMLSTARLET=xmlstarlet 2024-03-24T23:57:22.005 INFO:tasks.workunit.client.0.smithi080.stderr:+++ uname 2024-03-24T23:57:22.005 INFO:tasks.workunit.client.0.smithi080.stderr:++ '[' Linux = FreeBSD ']' 2024-03-24T23:57:22.005 INFO:tasks.workunit.client.0.smithi080.stderr:++ SED=sed 2024-03-24T23:57:22.005 INFO:tasks.workunit.client.0.smithi080.stderr:++ AWK=awk 2024-03-24T23:57:22.006 INFO:tasks.workunit.client.0.smithi080.stderr:+++ head -1 2024-03-24T23:57:22.006 INFO:tasks.workunit.client.0.smithi080.stderr:+++ stty -a 2024-03-24T23:57:22.006 INFO:tasks.workunit.client.0.smithi080.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-03-24T23:57:22.011 INFO:tasks.workunit.client.0.smithi080.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-24T23:57:22.011 INFO:tasks.workunit.client.0.smithi080.stderr:++ termwidth= 2024-03-24T23:57:22.011 INFO:tasks.workunit.client.0.smithi080.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-03-24T23:57:22.011 INFO:tasks.workunit.client.0.smithi080.stderr:++ DIFFCOLOPTS='-y ' 2024-03-24T23:57:22.011 INFO:tasks.workunit.client.0.smithi080.stderr:++ KERNCORE=kernel.core_pattern 2024-03-24T23:57:22.011 INFO:tasks.workunit.client.0.smithi080.stderr:++ EXTRA_OPTS= 2024-03-24T23:57:22.014 INFO:tasks.workunit.client.0.smithi080.stderr:++ test '' = TESTS 2024-03-24T23:57:22.015 INFO:tasks.workunit.client.0.smithi080.stderr:++ uname -m 2024-03-24T23:57:22.016 INFO:tasks.workunit.client.0.smithi080.stderr:+ arch=x86_64 2024-03-24T23:57:22.016 INFO:tasks.workunit.client.0.smithi080.stderr:+ case $arch in 2024-03-24T23:57:22.016 INFO:tasks.workunit.client.0.smithi080.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2024-03-24T23:57:22.016 INFO:tasks.workunit.client.0.smithi080.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:+ plugins=(jerasure shec lrc isa) 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:+ main test-erasure-code-plugins 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:+ local dir=td/test-erasure-code-plugins 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:+ shift 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:+ shopt -s -o xtrace 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-24T23:57:22.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-24T23:57:22.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-24T23:57:22.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-code-plugins 2024-03-24T23:57:22.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2024-03-24T23:57:22.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2024-03-24T23:57:22.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2024-03-24T23:57:22.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2024-03-24T23:57:22.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2024-03-24T23:57:22.018 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2024-03-24T23:57:22.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none ' 2024-03-24T23:57:22.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2024-03-24T23:57:22.019 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2024-03-24T23:57:22.019 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-24T23:57:22.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_preload_no_warning 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_preload_no_warning_default 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_preload_warning' 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-24T23:57:22.021 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-24T23:57:22.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-24T23:57:22.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-24T23:57:22.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-24T23:57:22.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-24T23:57:22.022 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-24T23:57:22.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-24T23:57:22.023 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-24T23:57:22.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-24T23:57:22.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-24T23:57:22.025 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-24T23:57:22.025 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T23:57:22.025 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-24T23:57:22.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-24T23:57:22.026 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T23:57:22.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-24T23:57:22.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-24T23:57:22.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-24T23:57:22.029 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-24T23:57:22.029 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:57:22.029 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:57:22.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-24T23:57:22.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-24T23:57:22.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-24T23:57:22.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-24T23:57:22.035 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-24T23:57:22.035 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:57:22.035 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:57:22.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-24T23:57:22.035 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-24T23:57:22.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-24T23:57:22.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-24T23:57:22.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-24T23:57:22.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-24T23:57:22.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2024-03-24T23:57:22.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-24T23:57:22.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-24T23:57:22.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-24T23:57:22.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-24T23:57:22.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-24T23:57:22.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-03-24T23:57:22.617 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-24T23:57:22.617 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-24T23:57:22.618 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-24T23:57:22.618 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-24T23:57:22.618 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:57:22.618 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:57:22.619 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:57:22.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-24T23:57:22.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-24T23:57:22.841 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-24T23:57:22.841 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-24T23:57:22.841 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-24T23:57:22.841 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-24T23:57:22.842 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-24T23:57:22.842 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-24T23:57:22.842 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-24T23:57:22.842 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-24T23:57:22.842 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-24T23:57:22.842 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:57:22.842 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:57:22.842 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-24T23:57:22.843 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-24T23:57:22.843 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-24T23:57:22.960 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-24T23:57:22.960 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-24T23:57:22.961 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-24T23:57:22.961 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-24T23:57:22.962 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-24T23:57:22.962 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-24T23:57:22.962 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-24T23:57:22.962 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-24T23:57:22.962 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-24T23:57:22.962 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:57:22.962 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:57:22.963 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-24T23:57:22.963 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-24T23:57:22.963 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-24T23:57:23.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2024-03-24T23:57:23.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-24T23:57:23.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-24T23:57:23.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-24T23:57:23.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-24T23:57:23.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-24T23:57:23.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-24T23:57:23.991 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-24T23:57:23.991 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-24T23:57:23.991 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-24T23:57:23.992 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-24T23:57:23.992 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:57:23.992 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:57:23.992 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:57:23.992 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-24T23:57:24.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-24T23:57:24.156 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2024-03-24T23:57:24.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-03-24T23:57:24.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2024-03-24T23:57:24.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-24T23:57:24.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-24T23:57:24.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-24T23:57:24.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-24T23:57:24.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-24T23:57:24.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-24T23:57:24.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-24T23:57:24.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:57:24.161 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-24T23:57:24.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-24T23:57:24.163 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-24T23:57:24.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=224b3b6d-d654-4f56-8561-770eed729f08 2024-03-24T23:57:24.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 224b3b6d-d654-4f56-8561-770eed729f08' 2024-03-24T23:57:24.164 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 224b3b6d-d654-4f56-8561-770eed729f08 2024-03-24T23:57:24.164 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-24T23:57:24.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkvQBmXVTvChAAqZSunN86NFOzpc+pudAjbg== 2024-03-24T23:57:24.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkvQBmXVTvChAAqZSunN86NFOzpc+pudAjbg=="}' 2024-03-24T23:57:24.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 224b3b6d-d654-4f56-8561-770eed729f08 -i td/test-erasure-code-plugins/0/new.json 2024-03-24T23:57:24.769 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-24T23:57:24.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-24T23:57:24.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkvQBmXVTvChAAqZSunN86NFOzpc+pudAjbg== --osd-uuid 224b3b6d-d654-4f56-8561-770eed729f08 2024-03-24T23:57:24.905 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:24.905+0000 7f9c205307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:57:24.937 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:24.937+0000 7f9c205307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:57:24.940 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:24.941+0000 7f9c205307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:57:24.940 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:24.941+0000 7f9c205307c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-24T23:57:24.940 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:24.941+0000 7f9c205307c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-24T23:57:24.940 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:24.941+0000 7f9c205307c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-24T23:57:25.031 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:25.029+0000 7f9c205307c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-24T23:57:31.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-24T23:57:31.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-24T23:57:31.576 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-24T23:57:31.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-24T23:57:31.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-24T23:57:31.812 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-24T23:57:31.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-24T23:57:31.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-24T23:57:31.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-24T23:57:31.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-24T23:57:31.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-24T23:57:31.837 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:31.837+0000 7f68e08b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:57:31.847 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:31.845+0000 7f68e08b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:57:31.848 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:31.849+0000 7f68e08b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:57:32.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-24T23:57:32.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-24T23:57:32.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-24T23:57:32.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-24T23:57:32.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-24T23:57:32.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:32.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-24T23:57:32.039 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-24T23:57:32.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:32.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:32.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:32.421 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:32.421+0000 7f68e08b17c0 -1 Falling back to public interface 2024-03-24T23:57:33.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:33.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:33.275 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-24T23:57:33.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-24T23:57:33.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:33.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:33.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:34.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:34.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:34.735 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-24T23:57:34.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-24T23:57:34.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:34.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:35.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:35.233 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:57:35.233+0000 7f68e08b17c0 -1 osd.0 0 log_to_monitors true 2024-03-24T23:57:36.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:36.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:36.153 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-24T23:57:36.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-24T23:57:36.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:36.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:36.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:37.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:37.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:37.650 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-24T23:57:37.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-24T23:57:37.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:37.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:38.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:39.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:39.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:39.255 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-24T23:57:39.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-24T23:57:39.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:39.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:39.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:40.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:40.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:40.675 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-24T23:57:40.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-24T23:57:40.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:40.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:41.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:42.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:42.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:42.162 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-24T23:57:42.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-24T23:57:42.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:42.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:42.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:43.636 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:43.636 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:43.636 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-24T23:57:43.636 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-24T23:57:43.637 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:43.637 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:44.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:45.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:45.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:45.154 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-24T23:57:45.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-24T23:57:45.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:45.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:45.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:46.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:46.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:46.608 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-24T23:57:46.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-24T23:57:46.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:46.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:47.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:48.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:48.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:48.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-24T23:57:48.180 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-24T23:57:48.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:48.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:48.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:49.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:49.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:49.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-24T23:57:49.653 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-24T23:57:49.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:49.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:50.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:51.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:51.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:51.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-24T23:57:51.089 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-24T23:57:51.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:51.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:51.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:52.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:52.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:52.926 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-24T23:57:52.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-24T23:57:52.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:52.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:54.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:55.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:55.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:55.147 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-24T23:57:55.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-24T23:57:55.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:55.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:56.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:57:57.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:57:57.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:57:57.758 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-24T23:57:57.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-24T23:57:57.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:57:57.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:57:59.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:00.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:00.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:00.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-24T23:58:00.402 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-24T23:58:00.403 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:00.403 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:58:01.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:02.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:02.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:02.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-24T23:58:02.153 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-24T23:58:02.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:02.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:58:02.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:03.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:03.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:03.765 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-24T23:58:03.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-24T23:58:03.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:03.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:58:04.179 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3112549552,v1:127.0.0.1:6801/3112549552] [v2:127.0.0.1:6802/3112549552,v1:127.0.0.1:6803/3112549552] exists,up 224b3b6d-d654-4f56-8561-770eed729f08 2024-03-24T23:58:04.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-24T23:58:04.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-24T23:58:04.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-24T23:58:04.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/1 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2024-03-24T23:58:04.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-24T23:58:04.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-24T23:58:04.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-24T23:58:04.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-24T23:58:04.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-24T23:58:04.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-24T23:58:04.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-24T23:58:04.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-24T23:58:04.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-24T23:58:04.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/1 2024-03-24T23:58:04.186 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-24T23:58:04.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=10701a28-e9ee-4498-b297-80e0f4280fd3 2024-03-24T23:58:04.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 10701a28-e9ee-4498-b297-80e0f4280fd3' 2024-03-24T23:58:04.187 INFO:tasks.workunit.client.0.smithi080.stdout:add osd1 10701a28-e9ee-4498-b297-80e0f4280fd3 2024-03-24T23:58:04.187 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-24T23:58:04.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMvgBmqKpxDBAAjq0ctk3fxWOh6Nt7KB28/w== 2024-03-24T23:58:04.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMvgBmqKpxDBAAjq0ctk3fxWOh6Nt7KB28/w=="}' 2024-03-24T23:58:04.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 10701a28-e9ee-4498-b297-80e0f4280fd3 -i td/test-erasure-code-plugins/1/new.json 2024-03-24T23:58:04.737 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-24T23:58:04.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/1/new.json 2024-03-24T23:58:04.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMvgBmqKpxDBAAjq0ctk3fxWOh6Nt7KB28/w== --osd-uuid 10701a28-e9ee-4498-b297-80e0f4280fd3 2024-03-24T23:58:04.778 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:04.777+0000 7ff7700b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:04.780 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:04.781+0000 7ff7700b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:04.781 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:04.781+0000 7ff7700b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:04.782 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:04.781+0000 7ff7700b27c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-03-24T23:58:04.782 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:04.781+0000 7ff7700b27c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-03-24T23:58:04.782 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:04.781+0000 7ff7700b27c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-03-24T23:58:04.837 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:04.837+0000 7ff7700b27c0 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2024-03-24T23:58:11.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2024-03-24T23:58:11.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-24T23:58:11.943 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd1 key to auth repository 2024-03-24T23:58:11.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-24T23:58:11.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-24T23:58:12.490 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.1 2024-03-24T23:58:12.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-24T23:58:12.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-24T23:58:12.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-24T23:58:12.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-24T23:58:12.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-24T23:58:12.520 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:12.524+0000 7f27a2d547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:12.522 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:12.524+0000 7f27a2d547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:12.523 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:12.528+0000 7f27a2d547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:12.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-24T23:58:12.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-24T23:58:12.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-24T23:58:12.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-24T23:58:12.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-24T23:58:12.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:12.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-24T23:58:12.929 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-24T23:58:12.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:12.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:13.093 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:13.096+0000 7f27a2d547c0 -1 Falling back to public interface 2024-03-24T23:58:13.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:14.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:14.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:14.366 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-24T23:58:14.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-24T23:58:14.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:14.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:14.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:15.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:15.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:15.764 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-24T23:58:15.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-24T23:58:15.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:15.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:15.828 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:15.832+0000 7f27a2d547c0 -1 osd.1 0 log_to_monitors true 2024-03-24T23:58:16.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:17.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:17.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:17.203 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-24T23:58:17.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-24T23:58:17.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:17.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:17.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:18.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:18.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:18.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-24T23:58:18.662 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-24T23:58:18.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:18.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:19.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:20.116 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:20.116 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:20.116 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-24T23:58:20.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-24T23:58:20.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:20.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:20.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:21.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:21.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:21.577 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-24T23:58:21.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-24T23:58:21.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:21.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:22.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:23.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:23.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:23.037 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-24T23:58:23.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-24T23:58:23.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:23.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:23.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:24.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:24.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:24.612 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-24T23:58:24.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-24T23:58:24.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:24.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:25.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:26.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:26.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:26.096 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-24T23:58:26.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-24T23:58:26.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:26.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:26.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:27.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:27.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:27.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-24T23:58:27.557 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-24T23:58:27.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:27.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:28.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:29.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:29.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:29.115 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-24T23:58:29.116 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-24T23:58:29.116 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:29.116 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:29.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:30.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:30.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:30.706 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-24T23:58:30.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-24T23:58:30.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:30.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:31.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:32.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:32.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:32.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-24T23:58:32.479 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-24T23:58:32.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:32.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:33.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:34.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:34.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:34.282 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-24T23:58:34.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-24T23:58:34.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:34.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:35.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:36.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:36.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:36.165 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-24T23:58:36.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-24T23:58:36.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:36.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:36.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:37.978 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:37.978 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:37.978 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-24T23:58:37.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-24T23:58:37.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:37.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:38.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:39.848 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:39.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:39.849 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-24T23:58:39.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-24T23:58:39.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:39.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:40.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:41.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:41.932 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-24T23:58:41.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:41.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-24T23:58:41.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:41.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:42.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:43.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:43.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:43.527 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-24T23:58:43.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-24T23:58:43.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:43.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-24T23:58:43.931 INFO:tasks.workunit.client.0.smithi080.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1289135762,v1:127.0.0.1:6811/1289135762] [v2:127.0.0.1:6812/1289135762,v1:127.0.0.1:6813/1289135762] exists,up 10701a28-e9ee-4498-b297-80e0f4280fd3 2024-03-24T23:58:43.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-24T23:58:43.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-24T23:58:43.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-24T23:58:43.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-03-24T23:58:43.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2024-03-24T23:58:43.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-24T23:58:43.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-24T23:58:43.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/2 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-24T23:58:43.936 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-24T23:58:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-24T23:58:43.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-24T23:58:43.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-24T23:58:43.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-24T23:58:43.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-24T23:58:43.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/2 2024-03-24T23:58:43.938 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-24T23:58:43.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=170b0cdf-6da3-4b97-aa43-c018b4bc0114 2024-03-24T23:58:43.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 170b0cdf-6da3-4b97-aa43-c018b4bc0114' 2024-03-24T23:58:43.939 INFO:tasks.workunit.client.0.smithi080.stdout:add osd2 170b0cdf-6da3-4b97-aa43-c018b4bc0114 2024-03-24T23:58:43.939 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-24T23:58:43.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAzvgBmbxs0ORAAgdTeHxQz3bJn2DdTEzwCKg== 2024-03-24T23:58:43.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAzvgBmbxs0ORAAgdTeHxQz3bJn2DdTEzwCKg=="}' 2024-03-24T23:58:43.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 170b0cdf-6da3-4b97-aa43-c018b4bc0114 -i td/test-erasure-code-plugins/2/new.json 2024-03-24T23:58:44.444 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-24T23:58:44.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/2/new.json 2024-03-24T23:58:44.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzvgBmbxs0ORAAgdTeHxQz3bJn2DdTEzwCKg== --osd-uuid 170b0cdf-6da3-4b97-aa43-c018b4bc0114 2024-03-24T23:58:44.493 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:44.495+0000 7f163bccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:44.514 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:44.515+0000 7f163bccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:44.515 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:44.515+0000 7f163bccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:44.516 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:44.519+0000 7f163bccb7c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-03-24T23:58:44.516 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:44.519+0000 7f163bccb7c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-03-24T23:58:44.516 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:44.519+0000 7f163bccb7c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-03-24T23:58:44.553 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:44.555+0000 7f163bccb7c0 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2024-03-24T23:58:51.539 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2024-03-24T23:58:51.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-24T23:58:51.541 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd2 key to auth repository 2024-03-24T23:58:51.541 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-24T23:58:51.541 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-24T23:58:52.070 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-24T23:58:52.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-24T23:58:52.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-24T23:58:52.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-24T23:58:52.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-24T23:58:52.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-24T23:58:52.099 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:52.099+0000 7f17644137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:52.101 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:52.103+0000 7f17644137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:52.102 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:52.103+0000 7f17644137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:58:52.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-24T23:58:52.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-24T23:58:52.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-24T23:58:52.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-24T23:58:52.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-24T23:58:52.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:52.486 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-24T23:58:52.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-24T23:58:52.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:52.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:58:52.670 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:52.671+0000 7f17644137c0 -1 Falling back to public interface 2024-03-24T23:58:52.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:53.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:53.921 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:53.921 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-24T23:58:53.921 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-24T23:58:53.921 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:53.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:58:54.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:55.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:55.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:55.319 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-24T23:58:55.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-24T23:58:55.320 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:55.320 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:58:55.323 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:58:55.323+0000 7f17644137c0 -1 osd.2 0 log_to_monitors true 2024-03-24T23:58:55.776 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:56.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:56.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:56.778 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-24T23:58:56.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-24T23:58:56.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:56.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:58:57.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:58.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:58.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:58.351 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-24T23:58:58.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-24T23:58:58.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:58.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:58:58.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:58:59.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:58:59.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:58:59.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-24T23:58:59.750 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-24T23:58:59.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:58:59.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:00.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:01.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:01.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:01.297 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-24T23:59:01.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-24T23:59:01.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:01.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:01.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:02.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:02.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:02.771 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-24T23:59:02.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-24T23:59:02.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:02.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:03.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:04.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:04.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:04.196 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-24T23:59:04.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-24T23:59:04.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:04.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:04.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:05.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:05.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:05.652 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-24T23:59:05.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-24T23:59:05.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:05.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:06.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:07.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:07.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:07.106 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-24T23:59:07.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-24T23:59:07.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:07.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:07.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:08.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:08.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:08.601 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-24T23:59:08.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-24T23:59:08.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:08.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:09.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:10.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:10.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:10.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-24T23:59:10.060 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-24T23:59:10.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:10.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:11.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:12.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:12.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:12.012 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-24T23:59:12.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-24T23:59:12.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:12.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:12.777 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:13.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:13.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:13.779 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-24T23:59:13.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-24T23:59:13.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:13.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:14.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:15.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:15.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:15.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-24T23:59:15.582 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-24T23:59:15.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:15.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:17.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:17.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:17.732 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-24T23:59:17.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-24T23:59:17.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:17.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:18.135 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:19.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:19.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:19.137 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-24T23:59:19.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-24T23:59:19.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:19.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:20.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:21.260 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:21.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:21.261 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-24T23:59:21.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-24T23:59:21.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:21.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:21.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:22.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:22.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:22.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-24T23:59:22.661 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-24T23:59:22.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:22.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-24T23:59:23.069 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2544761782,v1:127.0.0.1:6819/2544761782] [v2:127.0.0.1:6820/2544761782,v1:127.0.0.1:6821/2544761782] exists,up 170b0cdf-6da3-4b97-aa43-c018b4bc0114 2024-03-24T23:59:23.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-24T23:59:23.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-24T23:59:23.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-24T23:59:23.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2024-03-24T23:59:23.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-24T23:59:23.530 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' does not exist 2024-03-24T23:59:23.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-24T23:59:23.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-24T23:59:24.104 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' already exists 2024-03-24T23:59:24.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-24T23:59:25.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-24T23:59:27.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2024-03-24T23:59:27.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-24T23:59:27.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-24T23:59:27.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-24T23:59:27.773 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-24T23:59:27.774 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-24T23:59:27.774 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-24T23:59:27.774 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-24T23:59:27.774 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-24T23:59:27.774 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-24T23:59:27.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-24T23:59:27.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-24T23:59:27.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-24T23:59:27.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-24T23:59:27.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-24T23:59:27.892 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-24T23:59:28.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-24T23:59:28.362 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-24T23:59:28.362 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-24T23:59:28.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-24T23:59:28.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-24T23:59:28.363 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-24T23:59:28.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-24T23:59:28.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-24T23:59:28.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-24T23:59:28.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-24T23:59:28.557 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-24T23:59:28.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-24T23:59:28.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-24T23:59:28.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672978' 2024-03-24T23:59:28.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-24T23:59:28.698 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-24T23:59:28.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-24T23:59:28.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-24T23:59:28.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672978 2-64424509448' 2024-03-24T23:59:28.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-24T23:59:28.841 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-24T23:59:28.841 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-24T23:59:28.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-24T23:59:28.843 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-24T23:59:28.843 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-24T23:59:28.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-24T23:59:28.844 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 21474836504 2024-03-24T23:59:28.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-24T23:59:28.844 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-24T23:59:29.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-03-24T23:59:29.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-24T23:59:29.357 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-24T23:59:29.357 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-24T23:59:29.359 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-24T23:59:29.360 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-24T23:59:29.360 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-24T23:59:29.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-24T23:59:29.361 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 42949672978 2024-03-24T23:59:29.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-24T23:59:29.361 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-24T23:59:29.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672978 2024-03-24T23:59:29.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-24T23:59:30.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-24T23:59:30.790 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-24T23:59:31.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-24T23:59:31.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-24T23:59:31.191 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-24T23:59:31.192 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-24T23:59:31.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-24T23:59:31.193 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-24T23:59:31.194 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-24T23:59:31.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-24T23:59:31.195 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 64424509448 2024-03-24T23:59:31.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-24T23:59:31.195 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-24T23:59:31.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-03-24T23:59:31.724 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-24T23:59:31.724 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-24T23:59:31.725 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-24T23:59:32.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-24T23:59:32.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-24T23:59:32.237 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-24T23:59:32.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-24T23:59:32.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-24T23:59:32.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-24T23:59:32.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-24T23:59:32.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-24T23:59:32.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-24T23:59:32.643 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-24T23:59:32.644 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-24T23:59:32.644 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-24T23:59:33.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-24T23:59:33.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-24T23:59:33.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-24T23:59:33.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-03-24T23:59:33.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2024-03-24T23:59:33.710 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-03-24T23:59:33.710 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-24T23:59:33.711 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-24T23:59:33.711 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-24T23:59:33.711 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:33.711 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:33.711 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-24T23:59:33.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-03-24T23:59:33.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-24T23:59:33.810 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-24T23:59:33.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-03-24T23:59:33.830 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:33.457+0000 7fd6379bb640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-03-24T23:59:33.831 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:33.689+0000 7fd6379bb640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-03-24T23:59:33.831 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:33.689+0000 7fd6379bb640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-03-24T23:59:33.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-03-24T23:59:33.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2024-03-24T23:59:34.404 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-03-24T23:59:34.404 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-24T23:59:34.404 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-24T23:59:34.405 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-24T23:59:34.405 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:34.405 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:34.405 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-24T23:59:34.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-03-24T23:59:34.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-24T23:59:34.497 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-24T23:59:34.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-03-24T23:59:34.519 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:34.145+0000 7fd6379bb640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-03-24T23:59:34.519 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:34.385+0000 7fd6379bb640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-03-24T23:59:34.519 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:34.385+0000 7fd6379bb640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-03-24T23:59:34.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-03-24T23:59:34.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2024-03-24T23:59:35.277 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-03-24T23:59:35.277 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-24T23:59:35.277 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-24T23:59:35.277 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-24T23:59:35.277 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:35.277 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:35.277 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-24T23:59:35.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-03-24T23:59:35.278 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-24T23:59:35.379 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-24T23:59:35.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-03-24T23:59:35.400 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:34.841+0000 7fd6379bb640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-03-24T23:59:35.400 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:35.261+0000 7fd6379bb640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-03-24T23:59:35.400 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:35.261+0000 7fd6379bb640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-03-24T23:59:35.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-03-24T23:59:35.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2024-03-24T23:59:35.942 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-03-24T23:59:35.942 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-24T23:59:35.942 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-24T23:59:35.942 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-24T23:59:35.942 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:35.942 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:35.943 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-24T23:59:35.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-03-24T23:59:35.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-24T23:59:36.031 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-24T23:59:36.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-03-24T23:59:36.051 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:35.697+0000 7fd6379bb640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-03-24T23:59:36.051 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:35.921+0000 7fd6379bb640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-03-24T23:59:36.051 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:35.921+0000 7fd6379bb640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-03-24T23:59:36.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-03-24T23:59:36.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2024-03-24T23:59:36.612 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-03-24T23:59:36.612 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-24T23:59:36.612 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-24T23:59:36.612 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-24T23:59:36.613 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:36.613 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:36.613 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-24T23:59:36.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-03-24T23:59:36.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-24T23:59:36.710 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-24T23:59:36.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-03-24T23:59:36.731 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:36.357+0000 7fd6379bb640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-03-24T23:59:36.732 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:36.597+0000 7fd6379bb640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-03-24T23:59:36.732 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:36.597+0000 7fd6379bb640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-03-24T23:59:36.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-03-24T23:59:36.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2024-03-24T23:59:37.321 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-03-24T23:59:37.321 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-24T23:59:37.321 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-24T23:59:37.321 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-24T23:59:37.321 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:37.321 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:37.322 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-24T23:59:37.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-03-24T23:59:37.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-24T23:59:37.416 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-24T23:59:37.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-03-24T23:59:37.437 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:37.077+0000 7fd6379bb640 0 mon.a@0(leader).osd e30 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-03-24T23:59:37.437 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:37.301+0000 7fd6379bb640 0 mon.a@0(leader).osd e31 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-03-24T23:59:37.437 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-24T23:59:37.301+0000 7fd6379bb640 0 mon.a@0(leader).osd e31 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-03-24T23:59:37.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2024-03-24T23:59:37.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-24T23:59:37.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-24T23:59:37.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-24T23:59:37.439 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-24T23:59:37.439 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-24T23:59:37.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-24T23:59:37.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-24T23:59:37.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-24T23:59:37.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-24T23:59:37.749 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-24T23:59:37.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-24T23:59:37.750 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-24T23:59:37.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-24T23:59:37.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-24T23:59:37.752 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-24T23:59:37.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T23:59:37.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-24T23:59:37.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-24T23:59:37.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T23:59:37.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-24T23:59:37.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-24T23:59:37.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-24T23:59:37.778 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-24T23:59:37.778 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:37.779 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:37.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-24T23:59:37.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-24T23:59:37.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-24T23:59:37.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-03-24T23:59:37.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2024-03-24T23:59:37.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2024-03-24T23:59:37.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-03-24T23:59:37.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-03-24T23:59:37.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-24T23:59:37.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-24T23:59:37.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-24T23:59:37.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-24T23:59:37.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-24T23:59:37.780 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-24T23:59:37.780 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-24T23:59:37.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-24T23:59:37.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-24T23:59:37.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-24T23:59:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-24T23:59:37.782 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-24T23:59:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-24T23:59:37.782 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-24T23:59:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-24T23:59:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-24T23:59:37.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-24T23:59:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T23:59:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-24T23:59:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-24T23:59:37.785 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T23:59:37.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-24T23:59:37.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-24T23:59:37.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-24T23:59:37.789 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-24T23:59:37.789 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:37.789 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:37.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-24T23:59:37.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-24T23:59:37.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-24T23:59:37.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-24T23:59:37.792 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-24T23:59:37.792 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:37.792 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:37.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-24T23:59:37.795 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-24T23:59:37.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-24T23:59:37.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-24T23:59:37.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-24T23:59:37.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2024-03-24T23:59:37.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-24T23:59:37.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-24T23:59:37.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-24T23:59:37.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-24T23:59:37.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-24T23:59:37.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2024-03-24T23:59:38.292 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-24T23:59:38.292 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-24T23:59:38.292 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-24T23:59:38.292 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-24T23:59:38.292 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:38.292 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:38.293 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:59:38.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2024-03-24T23:59:38.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-24T23:59:38.523 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-24T23:59:38.524 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-24T23:59:38.524 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-24T23:59:38.524 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-24T23:59:38.524 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-24T23:59:38.524 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-24T23:59:38.524 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-24T23:59:38.525 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-24T23:59:38.525 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-24T23:59:38.525 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:38.525 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:38.525 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-24T23:59:38.525 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-24T23:59:38.526 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-24T23:59:38.632 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-24T23:59:38.632 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-24T23:59:38.632 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-24T23:59:38.632 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-24T23:59:38.633 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-24T23:59:38.633 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-24T23:59:38.633 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-24T23:59:38.633 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-24T23:59:38.633 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-24T23:59:38.633 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:38.633 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:38.633 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-24T23:59:38.633 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-24T23:59:38.634 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-24T23:59:38.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-03-24T23:59:38.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-24T23:59:38.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-24T23:59:38.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-24T23:59:38.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-24T23:59:38.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-24T23:59:38.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-24T23:59:39.627 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-24T23:59:39.627 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-24T23:59:39.627 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-24T23:59:39.627 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-24T23:59:39.627 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:39.627 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:39.627 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:59:39.628 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-24T23:59:39.628 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-24T23:59:39.666 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-03-24T23:59:39.667 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-24T23:59:39.667 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-24T23:59:39.667 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-24T23:59:39.667 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:39.667 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:39.667 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-24T23:59:39.668 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-03-24T23:59:39.668 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-24T23:59:39.764 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-24T23:59:39.777 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-24T23:59:39.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-24T23:59:39.779 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-24T23:59:39.779 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-24T23:59:39.779 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-24T23:59:39.780 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-24T23:59:39.780 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-24T23:59:39.780 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-24T23:59:39.780 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:59:39.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-24T23:59:39.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-24T23:59:39.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-24T23:59:39.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-24T23:59:39.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-24T23:59:39.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-24T23:59:39.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-24T23:59:39.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-24T23:59:39.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-24T23:59:39.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-24T23:59:39.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-24T23:59:39.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2024-03-24T23:59:39.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-24T23:59:39.781 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-24T23:59:39.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=11355e14-6140-4e39-8521-f03c968655e5 2024-03-24T23:59:39.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 11355e14-6140-4e39-8521-f03c968655e5' 2024-03-24T23:59:39.783 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 11355e14-6140-4e39-8521-f03c968655e5 2024-03-24T23:59:39.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-24T23:59:39.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrvgBmACT4LxAAFJmALUc9lVSBf0OF34uKlw== 2024-03-24T23:59:39.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrvgBmACT4LxAAFJmALUc9lVSBf0OF34uKlw=="}' 2024-03-24T23:59:39.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 11355e14-6140-4e39-8521-f03c968655e5 -i td/test-erasure-code-plugins/0/new.json 2024-03-24T23:59:40.132 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-24T23:59:40.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-24T23:59:40.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQBrvgBmACT4LxAAFJmALUc9lVSBf0OF34uKlw== --osd-uuid 11355e14-6140-4e39-8521-f03c968655e5 2024-03-24T23:59:40.182 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:40.181+0000 7fb63db6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:59:40.209 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:40.209+0000 7fb63db6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:59:40.210 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:40.209+0000 7fb63db6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:59:40.210 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:40.209+0000 7fb63db6c7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-24T23:59:40.211 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:40.209+0000 7fb63db6c7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-24T23:59:40.211 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:40.209+0000 7fb63db6c7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-24T23:59:40.265 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:40.265+0000 7fb63db6c7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-24T23:59:46.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-24T23:59:46.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-24T23:59:46.840 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-24T23:59:46.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-24T23:59:46.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-24T23:59:47.402 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-24T23:59:47.403 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-24T23:59:47.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-24T23:59:47.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2024-03-24T23:59:47.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-24T23:59:47.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-24T23:59:47.433 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:47.433+0000 7ff8782b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:59:47.435 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:47.433+0000 7ff8782b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:59:47.436 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:47.433+0000 7ff8782b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-24T23:59:47.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-24T23:59:47.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-24T23:59:47.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-24T23:59:47.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-24T23:59:47.780 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-24T23:59:47.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-24T23:59:47.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:47.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-24T23:59:47.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:47.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:59:48.002 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:48.001+0000 7ff8782b77c0 -1 Falling back to public interface 2024-03-24T23:59:48.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:49.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:49.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:49.203 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-24T23:59:49.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-24T23:59:49.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:49.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:59:49.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:50.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:50.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:50.601 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-24T23:59:50.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-24T23:59:50.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:50.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:59:50.619 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-24T23:59:50.617+0000 7ff8782b77c0 -1 osd.0 0 log_to_monitors true 2024-03-24T23:59:51.135 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:52.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:52.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:52.137 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-24T23:59:52.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-24T23:59:52.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:52.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:59:52.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:53.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:53.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:53.666 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-24T23:59:53.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-24T23:59:53.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:53.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:59:54.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:55.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:55.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:55.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-24T23:59:55.129 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-24T23:59:55.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:55.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:59:55.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:56.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:56.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:56.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-24T23:59:56.582 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-24T23:59:56.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:56.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:59:57.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:58.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:58.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:58.020 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-24T23:59:58.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-24T23:59:58.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:58.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-24T23:59:58.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-24T23:59:59.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-24T23:59:59.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-24T23:59:59.576 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-24T23:59:59.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-24T23:59:59.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-24T23:59:59.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:00.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:01.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:01.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:01.067 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:00:01.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:00:01.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:01.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:01.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:02.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:02.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:02.609 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:00:02.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:00:02.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:02.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:03.006 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:04.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:04.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:04.009 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:00:04.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:00:04.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:04.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:04.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:05.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:05.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:05.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:00:05.404 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:00:05.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:05.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:06.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:07.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:07.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:07.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:00:07.479 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:00:07.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:07.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:08.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:09.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:09.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:09.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:00:09.301 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:00:09.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:09.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:10.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:11.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:11.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:11.197 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:00:11.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:00:11.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:11.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:11.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:12.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:12.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:12.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:00:12.998 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:00:12.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:12.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:13.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:14.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:14.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:14.742 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:00:14.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:00:14.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:14.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:15.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:16.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:16.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:16.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:00:16.864 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:00:16.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:16.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:17.316 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1575020187,v1:127.0.0.1:6803/1575020187] [v2:127.0.0.1:6804/1575020187,v1:127.0.0.1:6805/1575020187] exists,up 11355e14-6140-4e39-8521-f03c968655e5 2024-03-25T00:00:17.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:00:17.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:00:17.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:00:17.318 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-03-25T00:00:17.318 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:00:17.318 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:00:17.318 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:00:17.318 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:17.318 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:17.318 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-osd.0.asok 2024-03-25T00:00:17.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-03-25T00:00:17.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-osd.0.asok log flush 2024-03-25T00:00:17.429 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:00:17.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-03-25T00:00:17.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-03-25T00:00:17.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-03-25T00:00:17.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:17.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:00:17.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:00:17.449 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:00:17.449 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:00:17.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:00:17.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:00:17.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:00:17.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:00:17.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:00:17.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:00:17.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:00:17.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:00:17.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:00:17.559 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:00:17.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:00:17.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:00:17.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:00:17.562 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:00:17.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:00:17.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:00:17.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:00:17.569 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:00:17.570 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:17.570 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:17.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:00:17.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:00:17.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:00:17.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:00:17.573 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:00:17.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:00:17.574 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:00:17.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:00:17.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:00:17.575 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:00:17.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:00:17.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:00:17.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:00:17.576 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:00:17.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:00:17.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:00:17.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:00:17.579 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:00:17.579 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:17.579 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:17.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:00:17.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:00:17.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:00:17.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-25T00:00:17.580 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:00:17.580 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:17.581 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:17.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-25T00:00:17.582 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:00:17.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:00:17.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:00:17.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-25T00:00:17.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2024-03-25T00:00:17.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:17.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:00:17.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:00:17.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:00:17.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-25T00:00:17.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2024-03-25T00:00:18.189 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:00:18.189 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:00:18.189 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:00:18.190 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:00:18.190 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:18.190 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:18.190 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:00:18.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2024-03-25T00:00:18.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:00:18.426 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:00:18.426 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:00:18.426 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:00:18.427 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:00:18.427 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:00:18.428 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:00:18.428 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:00:18.428 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:00:18.428 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:00:18.428 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:18.428 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:18.428 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:00:18.428 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:00:18.428 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-25T00:00:18.540 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:00:18.541 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:00:18.541 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:00:18.541 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:00:18.542 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:00:18.542 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:00:18.542 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:00:18.542 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:00:18.542 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:00:18.542 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:18.542 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:18.542 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:00:18.543 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:00:18.543 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-25T00:00:18.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-03-25T00:00:18.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:18.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:00:18.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:00:18.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:00:18.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-25T00:00:18.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:00:19.530 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:00:19.530 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:00:19.530 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:00:19.530 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:00:19.530 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:19.530 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:19.530 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:00:19.531 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:00:19.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:00:19.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-03-25T00:00:19.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:00:19.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:00:19.565 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:00:19.565 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:19.566 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:19.566 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:00:19.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-03-25T00:00:19.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-25T00:00:19.662 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:00:19.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2024-03-25T00:00:19.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:19.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:00:19.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:00:19.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:00:19.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-25T00:00:19.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-25T00:00:19.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:00:19.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:00:19.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:00:19.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-25T00:00:19.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-25T00:00:19.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:00:19.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:00:19.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-25T00:00:19.678 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:00:19.679 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:00:19.679 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:00:19.679 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:00:19.679 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:19.679 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:19.681 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:00:19.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:00:19.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:00:19.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:00:19.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:00:19.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-25T00:00:19.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-25T00:00:19.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:00:19.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:00:19.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:00:19.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:00:19.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:00:19.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2024-03-25T00:00:19.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-25T00:00:19.683 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:00:19.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=02a2aab9-ae1c-4e82-a9cf-333888d48654 2024-03-25T00:00:19.684 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 02a2aab9-ae1c-4e82-a9cf-333888d48654 2024-03-25T00:00:19.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 02a2aab9-ae1c-4e82-a9cf-333888d48654' 2024-03-25T00:00:19.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:00:19.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTvgBmRlSSKRAAE8FQkFg0G2pwKJaOo7Yfhg== 2024-03-25T00:00:19.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTvgBmRlSSKRAAE8FQkFg0G2pwKJaOo7Yfhg=="}' 2024-03-25T00:00:19.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 02a2aab9-ae1c-4e82-a9cf-333888d48654 -i td/test-erasure-code-plugins/0/new.json 2024-03-25T00:00:19.987 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:00:20.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-25T00:00:20.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQCTvgBmRlSSKRAAE8FQkFg0G2pwKJaOo7Yfhg== --osd-uuid 02a2aab9-ae1c-4e82-a9cf-333888d48654 2024-03-25T00:00:20.039 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:20.040+0000 7f6db0b977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:00:20.056 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:20.056+0000 7f6db0b977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:00:20.057 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:20.060+0000 7f6db0b977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:00:20.057 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:20.060+0000 7f6db0b977c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:00:20.057 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:20.060+0000 7f6db0b977c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:00:20.057 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:20.060+0000 7f6db0b977c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:00:20.096 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:20.096+0000 7f6db0b977c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-25T00:00:26.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-25T00:00:26.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:00:26.485 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:00:26.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:00:26.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:00:27.038 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:00:27.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:00:27.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2024-03-25T00:00:27.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:00:27.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:00:27.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:00:27.068 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:27.068+0000 7fe84a16d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:00:27.070 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:27.072+0000 7fe84a16d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:00:27.071 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:27.072+0000 7fe84a16d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:00:27.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:00:27.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:00:27.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:00:27.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:00:27.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:00:27.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:27.433 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:00:27.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:00:27.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:27.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:27.646 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:27.648+0000 7fe84a16d7c0 -1 Falling back to public interface 2024-03-25T00:00:27.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:28.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:28.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:28.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:00:28.845 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:00:28.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:28.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:29.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:30.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:30.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:30.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:00:30.249 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:00:30.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:30.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:30.251 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:30.252+0000 7fe84a16d7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:00:30.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:31.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:31.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:31.816 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:00:31.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:00:31.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:31.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:32.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:33.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:33.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:33.369 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:00:33.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:00:33.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:33.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:33.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:34.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:34.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:34.796 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:00:34.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:00:34.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:34.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:35.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:36.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:36.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:36.316 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:00:36.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:00:36.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:36.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:36.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:37.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:37.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:37.784 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:00:37.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:00:37.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:37.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:38.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:39.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:39.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:39.262 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:00:39.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:00:39.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:39.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:39.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:40.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:40.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:40.733 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:00:40.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:00:40.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:40.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:41.240 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:42.242 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:42.242 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:42.242 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:00:42.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:00:42.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:42.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:42.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:43.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:43.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:43.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:00:43.673 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:00:43.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:43.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:44.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:45.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:45.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:45.171 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:00:45.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:00:45.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:45.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:45.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:46.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:46.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:46.782 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:00:46.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:00:46.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:46.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:47.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:48.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:48.709 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:00:48.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:48.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:00:48.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:48.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:49.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:50.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:50.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:50.452 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:00:50.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:00:50.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:50.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:51.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:52.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:52.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:52.411 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:00:52.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:00:52.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:52.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:53.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:54.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:54.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:54.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:00:54.119 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:00:54.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:54.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:55.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:00:56.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:00:56.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:00:56.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:00:56.288 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:00:56.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:00:56.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:00:56.751 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2809841162,v1:127.0.0.1:6803/2809841162] [v2:127.0.0.1:6804/2809841162,v1:127.0.0.1:6805/2809841162] exists,up 02a2aab9-ae1c-4e82-a9cf-333888d48654 2024-03-25T00:00:56.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:00:56.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:00:56.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:00:56.753 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-03-25T00:00:56.753 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:00:56.753 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:00:56.753 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:00:56.754 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:56.754 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:56.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-osd.0.asok 2024-03-25T00:00:56.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-03-25T00:00:56.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-osd.0.asok log flush 2024-03-25T00:00:56.853 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:00:56.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-03-25T00:00:56.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-03-25T00:00:56.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-03-25T00:00:56.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:56.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:00:56.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:00:56.870 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:00:56.870 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:00:56.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:00:56.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:00:56.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:00:57.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:00:57.180 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:00:57.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:00:57.182 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:00:57.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:00:57.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:00:57.183 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:00:57.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:00:57.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:00:57.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:00:57.185 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:00:57.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:00:57.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:00:57.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:00:57.195 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:00:57.195 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:57.195 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:57.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:00:57.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:00:57.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:00:57.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-03-25T00:00:57.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-03-25T00:00:57.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:57.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-25T00:00:57.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:57.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:00:57.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:00:57.198 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:00:57.199 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:00:57.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:00:57.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:00:57.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:00:57.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:00:57.200 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:00:57.200 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:00:57.201 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:00:57.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:00:57.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:00:57.202 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:00:57.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:00:57.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:00:57.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:00:57.204 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:00:57.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:00:57.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:00:57.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:00:57.207 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:00:57.207 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:57.208 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:57.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:00:57.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:00:57.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:00:57.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-25T00:00:57.210 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:00:57.210 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:57.210 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-25T00:00:57.213 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:00:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:00:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:00:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-25T00:00:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2024-03-25T00:00:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:00:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:00:57.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:00:57.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-25T00:00:57.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2024-03-25T00:00:57.669 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:00:57.737 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:00:57.737 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:00:57.737 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:00:57.737 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:57.738 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:57.738 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:00:57.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2024-03-25T00:00:57.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:00:57.855 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:00:57.855 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:00:57.855 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:00:57.855 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:00:57.855 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:00:57.856 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:00:57.856 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:00:57.856 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:00:57.856 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:00:57.856 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:57.856 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:57.856 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:00:57.857 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:00:57.857 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-25T00:00:57.967 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:00:57.967 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:00:57.967 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:00:57.967 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:00:57.969 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:00:57.969 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:00:57.969 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:00:57.969 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:00:57.969 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:00:57.969 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:57.969 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:57.969 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:00:57.969 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:00:57.969 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-25T00:00:58.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-03-25T00:00:58.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:58.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:00:58.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:00:58.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:00:58.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-25T00:00:58.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:00:59.000 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:00:59.000 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:00:59.000 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:00:59.001 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:00:59.001 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:59.001 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:59.001 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:00:59.001 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:00:59.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:00:59.036 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-03-25T00:00:59.036 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:00:59.036 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:00:59.036 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:00:59.036 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:59.036 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:59.036 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:00:59.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-03-25T00:00:59.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-25T00:00:59.133 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:00:59.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2024-03-25T00:00:59.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-25T00:00:59.148 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:00:59.149 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:00:59.149 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:00:59.149 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:00:59.149 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:00:59.149 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:00:59.149 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-25T00:00:59.151 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:00:59.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1a692454-da82-4481-883f-ddd3072892c0 2024-03-25T00:00:59.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1a692454-da82-4481-883f-ddd3072892c0' 2024-03-25T00:00:59.152 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 1a692454-da82-4481-883f-ddd3072892c0 2024-03-25T00:00:59.152 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:00:59.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7vgBmijQWChAApoJhYnDmNDeZhvGXlkrvjA== 2024-03-25T00:00:59.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7vgBmijQWChAApoJhYnDmNDeZhvGXlkrvjA=="}' 2024-03-25T00:00:59.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1a692454-da82-4481-883f-ddd3072892c0 -i td/test-erasure-code-plugins/0/new.json 2024-03-25T00:00:59.498 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:00:59.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-25T00:00:59.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQC7vgBmijQWChAApoJhYnDmNDeZhvGXlkrvjA== --osd-uuid 1a692454-da82-4481-883f-ddd3072892c0 2024-03-25T00:00:59.550 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:59.551+0000 7f511e0ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:00:59.574 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:59.575+0000 7f511e0ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:00:59.576 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:59.575+0000 7f511e0ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:00:59.576 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:59.579+0000 7f511e0ad7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:00:59.576 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:59.579+0000 7f511e0ad7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:00:59.577 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:59.579+0000 7f511e0ad7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:00:59.615 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:00:59.615+0000 7f511e0ad7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-25T00:01:06.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-25T00:01:06.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:01:06.250 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:01:06.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:01:06.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:01:06.826 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:01:06.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:01:06.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2024-03-25T00:01:06.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:01:06.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:01:06.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:01:06.857 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:06.859+0000 7fa6b77367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:01:06.892 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:06.891+0000 7fa6b77367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:01:06.893 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:06.895+0000 7fa6b77367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:01:07.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:01:07.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:01:07.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:01:07.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:01:07.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:01:07.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:07.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:01:07.313 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:01:07.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:07.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:07.490 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:07.491+0000 7fa6b77367c0 -1 Falling back to public interface 2024-03-25T00:01:07.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:08.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:08.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:08.754 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:01:08.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:01:08.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:08.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:09.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:10.149 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:10.151+0000 7fa6b77367c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:01:10.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:10.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:10.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:01:10.155 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:01:10.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:10.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:10.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:11.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:11.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:11.662 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:01:11.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:01:11.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:11.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:12.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:13.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:13.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:13.106 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:01:13.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:01:13.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:13.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:13.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:14.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:14.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:14.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:01:14.570 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:01:14.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:14.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:15.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:16.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:16.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:16.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:01:16.159 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:01:16.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:16.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:16.636 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:17.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:17.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:17.638 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:01:17.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:01:17.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:17.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:18.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:19.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:19.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:19.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:01:19.180 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:01:19.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:19.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:19.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:20.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:20.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:20.656 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:01:20.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:01:20.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:20.658 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:21.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:22.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:22.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:22.124 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:01:22.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:01:22.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:22.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:22.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:23.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:23.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:23.555 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:01:23.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:01:23.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:23.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:24.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:25.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:25.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:25.264 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:01:25.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:01:25.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:25.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:25.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:26.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:26.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:26.884 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:01:26.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:01:26.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:26.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:27.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:28.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:28.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:28.726 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:01:28.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:01:28.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:28.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:29.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:30.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:30.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:30.554 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:01:30.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:01:30.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:30.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:31.359 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:32.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:32.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:32.361 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:01:32.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:01:32.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:32.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:33.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:34.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:34.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:34.056 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:01:34.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:01:34.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:34.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:35.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:36.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:36.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:36.182 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:01:36.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:01:36.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:36.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:36.635 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2923026788,v1:127.0.0.1:6803/2923026788] [v2:127.0.0.1:6804/2923026788,v1:127.0.0.1:6805/2923026788] exists,up 1a692454-da82-4481-883f-ddd3072892c0 2024-03-25T00:01:36.637 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:01:36.637 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:01:36.637 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:01:36.637 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-03-25T00:01:36.637 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:01:36.637 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:01:36.637 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:01:36.637 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:01:36.638 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:01:36.638 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-osd.0.asok 2024-03-25T00:01:36.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-03-25T00:01:36.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-osd.0.asok log flush 2024-03-25T00:01:36.741 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:01:36.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-03-25T00:01:36.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-03-25T00:01:36.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-03-25T00:01:36.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:01:36.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:01:36.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:01:36.760 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:01:36.760 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:01:36.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:01:36.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:01:36.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:01:36.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:01:37.041 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:01:37.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:01:37.052 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:01:37.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:01:37.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:01:37.052 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:01:37.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:01:37.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:01:37.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:01:37.052 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:01:37.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:01:37.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:01:37.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:01:37.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:01:37.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:01:37.054 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:01:37.054 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:01:37.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:01:37.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:01:37.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:01:37.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:01:37.054 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:01:37.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:01:37.055 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:01:37.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:01:37.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:01:37.055 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:01:37.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:01:37.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:01:37.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:01:37.055 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:01:37.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:01:37.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-25T00:01:37.056 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:01:37.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:01:37.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:01:37.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-25T00:01:37.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2024-03-25T00:01:37.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-25T00:01:37.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:01:37.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:01:37.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:01:37.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-25T00:01:37.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2024-03-25T00:01:37.378 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:01:37.378 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:01:37.379 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:01:37.379 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:01:37.379 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:01:37.379 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:01:37.379 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:01:37.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2024-03-25T00:01:37.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:01:37.591 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:01:37.591 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:01:37.591 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:01:37.591 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:01:37.591 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:01:37.592 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:01:37.592 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:01:37.592 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:01:37.593 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:01:37.594 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:01:37.594 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:01:37.594 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:01:37.594 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:01:37.594 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-25T00:01:37.705 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:01:37.705 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:01:37.705 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:01:37.705 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:01:37.705 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:01:37.706 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:01:37.706 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:01:37.706 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:01:37.706 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:01:37.707 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:01:37.707 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:01:37.707 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:01:37.707 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:01:37.707 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-25T00:01:37.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-03-25T00:01:37.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-25T00:01:37.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:01:37.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:01:37.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:01:37.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-25T00:01:37.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:01:38.719 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:01:38.719 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:01:38.719 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:01:38.720 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:01:38.720 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:01:38.720 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:01:38.720 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:01:38.721 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:01:38.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:01:38.755 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-03-25T00:01:38.755 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:01:38.755 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:01:38.756 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:01:38.756 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:01:38.756 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:01:38.756 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:01:38.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-03-25T00:01:38.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-25T00:01:38.853 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:01:38.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2024-03-25T00:01:38.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-25T00:01:38.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:01:38.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:01:38.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:01:38.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-25T00:01:38.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-25T00:01:38.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:01:38.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:01:38.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:01:38.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-25T00:01:38.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-25T00:01:38.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:01:38.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:01:38.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-25T00:01:38.871 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:01:38.871 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:01:38.871 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:01:38.872 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:01:38.872 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:01:38.872 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:01:38.872 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:01:38.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:01:38.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:01:38.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:01:38.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:01:38.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-25T00:01:38.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-25T00:01:38.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:01:38.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:01:38.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:01:38.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:01:38.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:01:38.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2024-03-25T00:01:38.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-25T00:01:38.874 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:01:38.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2c0d2fb5-98c2-4b0b-b4b7-04024c3d5453 2024-03-25T00:01:38.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2c0d2fb5-98c2-4b0b-b4b7-04024c3d5453' 2024-03-25T00:01:38.875 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 2c0d2fb5-98c2-4b0b-b4b7-04024c3d5453 2024-03-25T00:01:38.875 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:01:38.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDivgBmPzgHNRAAWTbkPhU2NxAAAdO98LwjeA== 2024-03-25T00:01:38.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDivgBmPzgHNRAAWTbkPhU2NxAAAdO98LwjeA=="}' 2024-03-25T00:01:38.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2c0d2fb5-98c2-4b0b-b4b7-04024c3d5453 -i td/test-erasure-code-plugins/0/new.json 2024-03-25T00:01:39.184 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:01:39.200 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-25T00:01:39.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQDivgBmPzgHNRAAWTbkPhU2NxAAAdO98LwjeA== --osd-uuid 2c0d2fb5-98c2-4b0b-b4b7-04024c3d5453 2024-03-25T00:01:39.228 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:39.231+0000 7fef64a357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:01:39.253 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:39.255+0000 7fef64a357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:01:39.254 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:39.255+0000 7fef64a357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:01:39.254 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:39.255+0000 7fef64a357c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:01:39.255 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:39.255+0000 7fef64a357c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:01:39.255 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:39.255+0000 7fef64a357c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:01:39.285 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:39.287+0000 7fef64a357c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-25T00:01:46.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-25T00:01:46.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:01:46.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:01:46.101 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:01:46.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:01:46.639 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:01:46.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:01:46.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:01:46.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:01:46.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:01:46.641 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2024-03-25T00:01:46.669 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:46.670+0000 7fb925b367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:01:46.671 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:46.670+0000 7fb925b367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:01:46.672 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:46.674+0000 7fb925b367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:01:47.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:01:47.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:01:47.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:01:47.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:01:47.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:01:47.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:47.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:01:47.059 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:01:47.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:47.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:47.234 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:47.234+0000 7fb925b367c0 -1 Falling back to public interface 2024-03-25T00:01:47.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:48.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:48.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:48.508 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:01:48.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:01:48.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:48.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:48.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:49.888 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:01:49.886+0000 7fb925b367c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:01:49.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:49.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:49.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:01:49.917 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:01:49.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:49.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:50.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:51.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:51.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:51.569 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:01:51.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:01:51.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:51.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:52.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:53.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:53.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:53.037 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:01:53.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:01:53.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:53.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:53.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:54.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:54.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:54.476 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:01:54.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:01:54.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:54.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:54.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:55.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:55.921 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:55.921 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:01:55.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:01:55.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:55.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:56.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:57.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:57.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:57.336 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:01:57.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:01:57.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:57.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:57.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:01:58.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:01:58.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:01:58.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:01:58.878 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:01:58.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:01:58.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:01:59.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:00.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:00.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:00.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:02:00.332 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:02:00.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:00.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:00.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:01.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:01.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:01.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:02:01.919 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:02:01.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:01.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:02.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:03.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:03.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:03.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:02:03.368 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:02:03.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:03.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:04.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:05.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:05.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:05.237 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:02:05.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:02:05.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:05.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:05.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:06.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:06.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:06.850 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:02:06.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:02:06.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:06.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:07.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:08.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:08.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:08.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:02:08.744 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:02:08.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:08.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:09.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:10.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:10.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:10.578 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:02:10.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:02:10.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:10.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:11.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:12.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:12.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:12.442 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:02:12.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:02:12.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:12.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:13.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:14.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:14.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:14.230 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:02:14.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:02:14.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:14.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:14.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:15.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:15.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:15.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:02:15.938 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:02:15.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:15.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:16.381 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/89871354,v1:127.0.0.1:6803/89871354] [v2:127.0.0.1:6804/89871354,v1:127.0.0.1:6805/89871354] exists,up 2c0d2fb5-98c2-4b0b-b4b7-04024c3d5453 2024-03-25T00:02:16.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:02:16.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:02:16.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:02:16.383 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-03-25T00:02:16.383 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:02:16.383 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:02:16.383 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:02:16.383 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:16.383 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:16.383 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-osd.0.asok 2024-03-25T00:02:16.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-03-25T00:02:16.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-osd.0.asok log flush 2024-03-25T00:02:16.493 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:02:16.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-03-25T00:02:16.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-03-25T00:02:16.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-03-25T00:02:16.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:16.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:02:16.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:02:16.513 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:02:16.513 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:02:16.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:02:16.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:02:16.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:02:16.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:02:16.824 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:02:16.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:02:16.825 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:02:16.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:02:16.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:02:16.827 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:02:16.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:02:16.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:02:16.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:02:16.829 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:02:16.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:02:16.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:02:16.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:02:16.839 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:02:16.839 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:16.839 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:16.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:02:16.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:02:16.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:02:16.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2024-03-25T00:02:16.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-03-25T00:02:16.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2024-03-25T00:02:16.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:16.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2024-03-25T00:02:16.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:16.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-25T00:02:16.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:16.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:02:16.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:02:16.844 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:02:16.844 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:02:16.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:02:16.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:02:16.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:02:16.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:02:16.845 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:02:16.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:02:16.846 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:02:16.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:02:16.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:02:16.848 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:02:16.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:02:16.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:02:16.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:02:16.849 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:02:16.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:02:16.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:02:16.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:02:16.854 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:02:16.854 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:16.854 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:16.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:02:16.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:02:16.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:02:16.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-25T00:02:16.858 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:02:16.858 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:16.858 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:16.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-25T00:02:16.859 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:02:16.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:02:16.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:02:16.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-25T00:02:16.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2024-03-25T00:02:16.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:16.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:02:16.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:02:16.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:02:16.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-25T00:02:16.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-03-25T00:02:17.331 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:02:17.331 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:02:17.331 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:02:17.332 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:02:17.332 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:17.332 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:17.332 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:02:17.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T00:02:17.541 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:02:17.544 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:02:17.544 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:02:17.544 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:02:17.544 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:02:17.544 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:02:17.544 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:02:17.545 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:02:17.545 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:02:17.545 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:02:17.545 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:17.545 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:17.545 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:02:17.545 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:02:17.545 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-25T00:02:17.660 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:02:17.660 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:02:17.660 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:02:17.661 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:02:17.662 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:02:17.662 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:02:17.662 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:02:17.662 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:02:17.662 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:02:17.662 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:17.662 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:17.662 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:02:17.662 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:02:17.663 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-25T00:02:17.771 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2024-03-25T00:02:17.772 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:02:17.772 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:02:17.772 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:02:17.773 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:17.773 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:17.773 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:02:17.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2024-03-25T00:02:17.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-25T00:02:17.870 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:02:17.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2024-03-25T00:02:17.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:17.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:02:17.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:02:17.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:02:17.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-25T00:02:17.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:02:18.806 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:02:18.806 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:02:18.806 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:02:18.806 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:02:18.806 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:18.806 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:18.807 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:02:18.807 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:02:18.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:02:18.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2024-03-25T00:02:18.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:18.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:02:18.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:02:18.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:02:18.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-25T00:02:18.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-25T00:02:18.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:02:18.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:02:18.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:02:18.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-25T00:02:18.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-25T00:02:18.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:02:18.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:02:18.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-25T00:02:18.841 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:02:18.842 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:02:18.842 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:02:18.842 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:02:18.842 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:18.842 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:18.842 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:02:18.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:02:18.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:02:18.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:02:18.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:02:18.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-25T00:02:18.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-25T00:02:18.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:02:18.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:02:18.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:02:18.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:02:18.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:02:18.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:02:18.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-25T00:02:18.843 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:02:18.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62dac9f2-7b6b-4b9d-be65-d5759c9177f0 2024-03-25T00:02:18.844 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 62dac9f2-7b6b-4b9d-be65-d5759c9177f0 2024-03-25T00:02:18.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 62dac9f2-7b6b-4b9d-be65-d5759c9177f0' 2024-03-25T00:02:18.845 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:02:18.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKvwBmQrpaMxAAHUwCcXcmO9h7QVIGFszPCg== 2024-03-25T00:02:18.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKvwBmQrpaMxAAHUwCcXcmO9h7QVIGFszPCg=="}' 2024-03-25T00:02:18.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62dac9f2-7b6b-4b9d-be65-d5759c9177f0 -i td/test-erasure-code-plugins/0/new.json 2024-03-25T00:02:19.180 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:02:19.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-25T00:02:19.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKvwBmQrpaMxAAHUwCcXcmO9h7QVIGFszPCg== --osd-uuid 62dac9f2-7b6b-4b9d-be65-d5759c9177f0 2024-03-25T00:02:19.225 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:19.226+0000 7fe1ec86d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:02:19.257 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:19.258+0000 7fe1ec86d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:02:19.258 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:19.258+0000 7fe1ec86d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:02:19.258 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:19.258+0000 7fe1ec86d7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:02:19.258 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:19.258+0000 7fe1ec86d7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:02:19.259 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:19.258+0000 7fe1ec86d7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:02:19.297 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:19.298+0000 7fe1ec86d7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-25T00:02:25.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-25T00:02:25.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:02:25.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:02:25.816 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:02:25.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:02:26.351 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:02:26.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:02:26.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:02:26.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:02:26.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:02:26.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:02:26.380 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:26.382+0000 7f84c65207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:02:26.382 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:26.382+0000 7f84c65207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:02:26.383 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:26.382+0000 7f84c65207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:02:26.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:02:26.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:02:26.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:02:26.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:02:26.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:02:26.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:26.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:02:26.754 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:02:26.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:26.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:26.929 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:26.930+0000 7f84c65207c0 -1 Falling back to public interface 2024-03-25T00:02:27.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:28.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:28.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:28.147 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:02:28.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:02:28.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:28.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:28.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:29.545 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:29.546+0000 7f84c65207c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:02:29.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:29.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:29.568 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:02:29.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:02:29.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:29.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:30.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:31.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:31.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:31.108 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:02:31.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:02:31.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:31.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:31.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:32.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:32.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:32.559 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:02:32.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:02:32.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:32.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:33.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:34.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:34.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:34.016 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:02:34.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:02:34.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:34.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:34.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:35.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:35.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:35.625 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:02:35.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:02:35.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:35.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:36.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:37.074 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:37.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:37.075 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:02:37.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:02:37.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:37.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:37.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:38.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:38.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:38.580 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:02:38.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:02:38.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:38.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:39.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:40.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:40.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:40.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:02:40.029 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:02:40.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:40.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:40.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:41.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:41.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:41.605 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:02:41.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:02:41.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:41.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:42.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:43.025 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:43.025 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:43.025 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:02:43.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:02:43.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:43.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:43.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:44.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:44.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:44.592 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:02:44.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:02:44.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:44.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:45.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:46.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:46.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:46.205 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:02:46.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:02:46.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:46.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:47.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:48.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:48.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:48.136 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:02:48.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:02:48.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:48.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:48.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:49.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:49.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:49.931 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:02:49.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:02:49.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:49.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:50.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:51.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:51.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:51.824 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:02:51.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:02:51.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:51.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:52.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:53.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:53.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:53.784 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:02:53.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:02:53.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:53.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:54.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:02:55.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:02:55.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:02:55.963 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:02:55.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:02:55.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:02:55.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:02:56.435 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3751586490,v1:127.0.0.1:6803/3751586490] [v2:127.0.0.1:6804/3751586490,v1:127.0.0.1:6805/3751586490] exists,up 62dac9f2-7b6b-4b9d-be65-d5759c9177f0 2024-03-25T00:02:56.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:02:56.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:02:56.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:02:56.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2024-03-25T00:02:56.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:02:56.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:02:56.437 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:02:56.437 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:56.437 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:56.438 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-osd.0.asok 2024-03-25T00:02:56.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2024-03-25T00:02:56.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-osd.0.asok log flush 2024-03-25T00:02:56.540 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:02:56.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2024-03-25T00:02:56.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2024-03-25T00:02:56.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2024-03-25T00:02:56.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:56.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:02:56.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:02:56.561 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:02:56.561 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:02:56.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:02:56.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:02:56.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:02:56.669 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:02:56.670 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:02:56.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:02:56.671 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:02:56.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:02:56.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:02:56.673 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:02:56.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:02:56.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:02:56.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:02:56.673 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:02:56.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:02:56.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:02:56.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:02:56.680 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:02:56.681 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:56.681 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:56.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:02:56.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:02:56.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:02:56.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2024-03-25T00:02:56.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-03-25T00:02:56.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2024-03-25T00:02:56.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:56.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-03-25T00:02:56.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-03-25T00:02:56.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:56.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-25T00:02:56.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:56.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:02:56.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:02:56.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:02:56.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:02:56.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:02:56.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:02:56.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:02:56.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:02:56.685 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:02:56.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:02:56.686 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:02:56.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:02:56.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:02:56.690 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:02:56.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:02:56.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:02:56.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:02:56.690 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:02:56.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:02:56.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:02:56.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:02:56.694 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:02:56.694 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:56.694 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:56.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:02:56.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:02:56.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:02:56.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-25T00:02:56.695 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:02:56.695 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:56.695 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:56.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-25T00:02:56.697 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:02:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:02:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:02:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-25T00:02:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2024-03-25T00:02:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:02:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:02:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:02:56.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-25T00:02:56.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2024-03-25T00:02:57.182 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:02:57.182 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:02:57.182 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:02:57.182 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:02:57.182 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:57.182 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:57.183 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:02:57.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2024-03-25T00:02:57.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:02:57.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:02:57.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:02:57.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:02:57.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:02:57.404 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:02:57.404 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:02:57.404 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:02:57.404 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:02:57.405 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:02:57.405 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:57.405 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:57.405 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:02:57.405 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:02:57.405 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-25T00:02:57.522 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:02:57.523 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:02:57.523 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:02:57.523 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:02:57.524 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:02:57.524 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:02:57.524 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:02:57.524 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:02:57.524 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:02:57.524 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:57.524 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:57.524 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:02:57.525 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:02:57.525 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-25T00:02:57.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-03-25T00:02:57.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:57.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:02:57.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:02:57.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:02:57.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-25T00:02:57.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:02:58.541 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:02:58.541 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:02:58.541 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:02:58.541 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:02:58.541 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:58.541 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:58.542 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:02:58.542 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:02:58.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:02:58.576 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-03-25T00:02:58.576 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:02:58.576 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:02:58.576 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:02:58.576 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:58.576 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:58.576 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:02:58.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:02:58.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-25T00:02:58.678 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:02:58.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2024-03-25T00:02:58.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-25T00:02:58.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:02:58.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:02:58.698 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:02:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:02:58.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:02:58.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:02:58.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2024-03-25T00:02:58.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-25T00:02:58.700 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:02:58.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8875438e-68a3-4f1f-b63f-a36fcba16030 2024-03-25T00:02:58.700 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 8875438e-68a3-4f1f-b63f-a36fcba16030 2024-03-25T00:02:58.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8875438e-68a3-4f1f-b63f-a36fcba16030' 2024-03-25T00:02:58.701 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:02:58.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAyvwBmq3TaKhAAdSane/woog8DdkVx1c59/w== 2024-03-25T00:02:58.717 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAyvwBmq3TaKhAAdSane/woog8DdkVx1c59/w=="}' 2024-03-25T00:02:58.717 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8875438e-68a3-4f1f-b63f-a36fcba16030 -i td/test-erasure-code-plugins/0/new.json 2024-03-25T00:02:59.022 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:02:59.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-25T00:02:59.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQAyvwBmq3TaKhAAdSane/woog8DdkVx1c59/w== --osd-uuid 8875438e-68a3-4f1f-b63f-a36fcba16030 2024-03-25T00:02:59.077 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:59.077+0000 7fdc6f7947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:02:59.099 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:59.097+0000 7fdc6f7947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:02:59.101 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:59.101+0000 7fdc6f7947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:02:59.101 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:59.101+0000 7fdc6f7947c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:02:59.102 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:59.101+0000 7fdc6f7947c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:02:59.102 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:59.101+0000 7fdc6f7947c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:02:59.130 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:02:59.129+0000 7fdc6f7947c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-25T00:03:05.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-25T00:03:05.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:03:05.738 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:03:05.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:03:05.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:03:06.298 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:03:06.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:03:06.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2024-03-25T00:03:06.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:03:06.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:03:06.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:03:06.328 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:06.329+0000 7ff26ae827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:03:06.331 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:06.329+0000 7ff26ae827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:03:06.332 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:06.333+0000 7ff26ae827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:03:06.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:03:06.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:03:06.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:03:06.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:03:06.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:03:06.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:06.746 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:03:06.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:03:06.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:06.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:06.889 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:06.889+0000 7ff26ae827c0 -1 Falling back to public interface 2024-03-25T00:03:07.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:08.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:08.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:08.156 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:03:08.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:03:08.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:08.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:08.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:09.529 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:09.529+0000 7ff26ae827c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:03:09.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:09.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:09.570 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:03:09.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:03:09.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:09.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:10.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:11.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:11.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:11.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:03:11.144 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:03:11.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:11.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:11.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:12.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:12.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:12.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:03:12.622 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:03:12.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:12.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:13.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:14.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:14.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:14.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:03:14.094 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:03:14.094 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:14.094 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:14.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:15.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:15.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:15.656 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:03:15.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:03:15.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:15.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:16.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:17.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:17.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:17.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:03:17.106 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:03:17.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:17.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:17.717 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:18.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:18.720 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:03:18.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:18.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:03:18.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:18.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:19.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:20.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:20.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:20.357 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:03:20.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:03:20.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:20.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:20.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:21.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:21.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:21.769 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:03:21.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:03:21.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:21.770 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:22.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:23.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:23.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:23.364 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:03:23.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:03:23.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:23.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:24.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:25.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:25.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:25.814 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:03:25.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:03:25.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:25.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:26.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:27.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:27.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:27.705 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:03:27.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:03:27.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:27.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:28.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:29.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:29.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:29.690 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:03:29.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:03:29.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:29.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:30.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:31.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:31.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:31.555 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:03:31.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:03:31.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:31.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:32.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:33.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:33.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:33.312 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:03:33.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:03:33.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:33.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:34.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:35.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:35.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:35.514 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:03:35.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:03:35.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:35.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:36.015 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2646593975,v1:127.0.0.1:6803/2646593975] [v2:127.0.0.1:6804/2646593975,v1:127.0.0.1:6805/2646593975] exists,up 8875438e-68a3-4f1f-b63f-a36fcba16030 2024-03-25T00:03:36.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:03:36.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:03:36.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:03:36.016 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-03-25T00:03:36.016 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:03:36.016 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:03:36.016 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:03:36.016 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:03:36.016 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:03:36.017 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-osd.0.asok 2024-03-25T00:03:36.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:03:36.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-osd.0.asok log flush 2024-03-25T00:03:36.116 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:03:36.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-03-25T00:03:36.131 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:02:57.221+0000 7f9adab60c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-03-25T00:03:36.131 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2024-03-25T00:03:36.132 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:03:06.889+0000 7ff26ae827c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-03-25T00:03:36.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-03-25T00:03:36.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:03:36.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:03:36.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:03:36.133 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:03:36.134 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:03:36.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:03:36.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:03:36.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:03:36.242 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:03:36.243 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:03:36.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:03:36.243 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:03:36.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:03:36.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:03:36.245 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:03:36.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:03:36.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:03:36.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:03:36.247 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:03:36.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:03:36.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:03:36.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:03:36.255 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:03:36.255 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:03:36.255 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:03:36.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:03:36.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:03:36.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:03:36.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-03-25T00:03:36.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-03-25T00:03:36.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-25T00:03:36.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-25T00:03:36.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:03:36.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:03:36.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:03:36.258 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:03:36.259 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:03:36.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:03:36.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:03:36.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:03:36.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:03:36.260 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:03:36.260 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:03:36.260 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:03:36.260 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:03:36.260 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:03:36.261 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:03:36.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:03:36.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:03:36.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:03:36.262 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:03:36.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:03:36.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:03:36.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:03:36.265 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:03:36.265 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:03:36.265 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:03:36.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:03:36.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:03:36.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:03:36.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-25T00:03:36.266 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:03:36.267 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:03:36.267 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:03:36.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-25T00:03:36.269 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:03:36.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:03:36.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:03:36.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-25T00:03:36.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2024-03-25T00:03:36.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-25T00:03:36.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:03:36.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:03:36.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:03:36.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-25T00:03:36.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2024-03-25T00:03:36.753 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:03:36.753 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:03:36.753 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:03:36.754 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:03:36.754 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:03:36.754 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:03:36.755 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:03:36.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2024-03-25T00:03:36.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:03:36.982 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:03:36.982 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:03:36.982 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:03:36.982 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:03:36.983 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:03:36.983 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:03:36.983 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:03:36.984 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:03:36.984 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:03:36.984 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:03:36.984 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:03:36.984 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:03:36.984 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:03:36.984 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-25T00:03:37.094 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:03:37.094 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:03:37.094 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:03:37.094 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:03:37.095 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:03:37.095 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:03:37.095 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:03:37.095 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:03:37.095 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:03:37.096 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:03:37.096 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:03:37.096 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:03:37.096 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:03:37.096 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-25T00:03:37.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-03-25T00:03:37.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-25T00:03:37.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:03:37.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:03:37.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:03:37.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-25T00:03:37.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:03:38.111 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:03:38.111 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:03:38.111 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:03:38.111 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:03:38.111 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:03:38.111 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:03:38.111 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:03:38.112 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:03:38.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:03:38.146 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-03-25T00:03:38.146 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:03:38.146 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:03:38.147 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:03:38.147 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:03:38.147 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:03:38.147 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:03:38.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:03:38.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-25T00:03:38.245 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:03:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2024-03-25T00:03:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-25T00:03:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:03:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:03:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:03:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-25T00:03:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-25T00:03:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:03:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:03:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:03:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-25T00:03:38.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-25T00:03:38.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:03:38.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:03:38.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-25T00:03:38.262 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:03:38.262 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:03:38.262 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:03:38.262 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:03:38.262 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:03:38.262 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:03:38.262 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:03:38.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-25T00:03:38.264 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:03:38.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7db8e852-254b-4392-8f2a-9ea7772d2773 2024-03-25T00:03:38.265 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 7db8e852-254b-4392-8f2a-9ea7772d2773 2024-03-25T00:03:38.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7db8e852-254b-4392-8f2a-9ea7772d2773' 2024-03-25T00:03:38.265 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:03:38.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBavwBm0woAERAAGhXX+4wfRMtgcdulMlxd5g== 2024-03-25T00:03:38.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBavwBm0woAERAAGhXX+4wfRMtgcdulMlxd5g=="}' 2024-03-25T00:03:38.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7db8e852-254b-4392-8f2a-9ea7772d2773 -i td/test-erasure-code-plugins/0/new.json 2024-03-25T00:03:38.594 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:03:38.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-25T00:03:38.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQBavwBm0woAERAAGhXX+4wfRMtgcdulMlxd5g== --osd-uuid 7db8e852-254b-4392-8f2a-9ea7772d2773 2024-03-25T00:03:38.647 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:38.645+0000 7fda3b8cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:03:38.671 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:38.669+0000 7fda3b8cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:03:38.672 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:38.673+0000 7fda3b8cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:03:38.673 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:38.673+0000 7fda3b8cd7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:03:38.673 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:38.673+0000 7fda3b8cd7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:03:38.673 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:38.673+0000 7fda3b8cd7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:03:38.727 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:38.725+0000 7fda3b8cd7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-25T00:03:45.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-25T00:03:45.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:03:45.381 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:03:45.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:03:45.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:03:45.929 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:03:45.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:03:45.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2024-03-25T00:03:45.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:03:45.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:03:45.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:03:45.953 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:45.953+0000 7f8c7bbeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:03:45.956 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:45.957+0000 7f8c7bbeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:03:45.957 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:45.957+0000 7f8c7bbeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:03:46.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:03:46.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:03:46.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:03:46.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:03:46.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:03:46.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:46.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:03:46.379 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:03:46.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:46.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:46.549 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:46.549+0000 7f8c7bbeb7c0 -1 Falling back to public interface 2024-03-25T00:03:46.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:47.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:47.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:47.832 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:03:47.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:03:47.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:47.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:48.234 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:49.212 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:03:49.216+0000 7f8c7bbeb7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:03:49.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:49.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:49.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:03:49.236 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:03:49.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:49.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:49.967 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:50.968 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:50.968 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:50.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:03:50.969 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:03:50.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:50.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:51.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:52.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:52.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:52.424 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:03:52.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:03:52.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:52.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:52.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:53.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:53.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:53.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:03:53.866 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:03:53.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:53.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:54.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:55.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:55.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:55.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:03:55.489 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:03:55.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:55.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:55.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:56.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:56.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:56.905 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:03:56.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:03:56.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:56.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:57.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:58.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:58.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:58.387 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:03:58.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:03:58.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:58.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:03:58.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:03:59.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:03:59.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:03:59.845 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:03:59.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:03:59.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:03:59.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:00.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:01.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:01.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:01.346 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:04:01.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:04:01.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:01.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:01.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:02.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:02.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:02.760 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:04:02.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:04:02.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:02.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:03.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:04.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:04.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:04.378 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:04:04.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:04:04.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:04.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:04.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:05.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:05.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:05.950 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:04:05.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:04:05.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:05.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:06.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:07.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:07.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:07.908 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:04:07.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:04:07.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:07.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:08.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:09.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:09.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:09.766 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:04:09.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:04:09.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:09.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:10.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:11.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:11.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:11.531 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:04:11.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:04:11.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:11.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:12.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:13.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:13.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:13.139 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:04:13.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:04:13.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:13.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:14.241 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:15.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:15.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:15.243 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:04:15.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:04:15.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:15.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:15.685 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3886646106,v1:127.0.0.1:6803/3886646106] [v2:127.0.0.1:6804/3886646106,v1:127.0.0.1:6805/3886646106] exists,up 7db8e852-254b-4392-8f2a-9ea7772d2773 2024-03-25T00:04:15.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:04:15.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:04:15.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:04:15.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-03-25T00:04:15.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:04:15.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:04:15.687 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:04:15.687 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:15.687 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:15.688 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-osd.0.asok 2024-03-25T00:04:15.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:04:15.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-osd.0.asok log flush 2024-03-25T00:04:15.798 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:04:15.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-03-25T00:04:15.814 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:03:36.797+0000 7f34a9016c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-03-25T00:04:15.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2024-03-25T00:04:15.816 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:03:46.549+0000 7f8c7bbeb7c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-03-25T00:04:15.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-03-25T00:04:15.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:15.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:04:15.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:04:15.817 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:04:15.817 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:04:15.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:04:15.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:04:15.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:04:15.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:04:15.925 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:04:15.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:04:15.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:04:15.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:04:15.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:04:15.928 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:04:15.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:04:15.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:04:15.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:04:15.930 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:04:15.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:04:15.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:04:15.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:04:15.938 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:04:15.938 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:15.938 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:15.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:04:15.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:04:15.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:04:15.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-03-25T00:04:15.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-03-25T00:04:15.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:15.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-25T00:04:15.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:15.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:04:15.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:04:15.941 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:04:15.941 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:04:15.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:04:15.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:04:15.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:04:15.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:04:15.942 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:04:15.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:04:15.943 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:04:15.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:04:15.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:04:15.945 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:04:15.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:04:15.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:04:15.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:04:15.946 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:04:15.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:04:15.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:04:15.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:04:15.948 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:04:15.948 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:15.948 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:15.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:04:15.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:04:15.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:04:15.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-25T00:04:15.951 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:04:15.951 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:15.951 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:15.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-25T00:04:15.953 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:04:15.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:04:15.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:04:15.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-25T00:04:15.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2024-03-25T00:04:15.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:15.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:04:15.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:04:15.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:04:15.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-25T00:04:15.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2024-03-25T00:04:16.462 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:04:16.462 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:04:16.462 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:04:16.463 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:04:16.463 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:16.463 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:16.463 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:04:16.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2024-03-25T00:04:16.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:04:16.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:04:16.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:04:16.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:04:16.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:04:16.685 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:04:16.685 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:04:16.685 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:04:16.685 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:04:16.686 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:04:16.686 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:16.686 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:16.686 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:04:16.686 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:04:16.686 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-25T00:04:16.805 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:04:16.805 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:04:16.805 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:04:16.805 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:04:16.806 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:04:16.806 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:04:16.806 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:04:16.806 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:04:16.807 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:04:16.808 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:16.808 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:16.808 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:04:16.808 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:04:16.808 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-25T00:04:16.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-03-25T00:04:16.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:16.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:04:16.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:04:16.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:04:16.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-25T00:04:16.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:04:17.831 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:04:17.910 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:04:17.910 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:04:17.910 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:04:17.910 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:17.910 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:17.910 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:04:17.912 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:04:17.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:04:17.912 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-03-25T00:04:17.912 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:04:17.912 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:04:17.912 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:04:17.912 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:17.912 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:17.912 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:04:17.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:04:17.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-25T00:04:18.007 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:04:18.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2024-03-25T00:04:18.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:18.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:04:18.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:04:18.028 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:04:18.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:04:18.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:04:18.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:04:18.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2024-03-25T00:04:18.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-25T00:04:18.032 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:04:18.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d1be9c4-aa04-4dca-8a48-6c648474c1cf 2024-03-25T00:04:18.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2d1be9c4-aa04-4dca-8a48-6c648474c1cf' 2024-03-25T00:04:18.033 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 2d1be9c4-aa04-4dca-8a48-6c648474c1cf 2024-03-25T00:04:18.034 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:04:18.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCvwBmsJ52AxAAgRO8HygHIQjF9FPtNC3vTA== 2024-03-25T00:04:18.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCvwBmsJ52AxAAgRO8HygHIQjF9FPtNC3vTA=="}' 2024-03-25T00:04:18.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d1be9c4-aa04-4dca-8a48-6c648474c1cf -i td/test-erasure-code-plugins/0/new.json 2024-03-25T00:04:18.377 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:04:18.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-25T00:04:18.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQCCvwBmsJ52AxAAgRO8HygHIQjF9FPtNC3vTA== --osd-uuid 2d1be9c4-aa04-4dca-8a48-6c648474c1cf 2024-03-25T00:04:18.422 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:18.424+0000 7f84c30c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:04:18.464 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:18.468+0000 7f84c30c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:04:18.466 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:18.468+0000 7f84c30c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:04:18.466 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:18.468+0000 7f84c30c97c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:04:18.466 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:18.468+0000 7f84c30c97c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:04:18.467 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:18.468+0000 7f84c30c97c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:04:18.511 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:18.516+0000 7f84c30c97c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-25T00:04:25.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-25T00:04:25.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:04:25.111 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:04:25.111 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:04:25.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:04:25.672 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:04:25.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:04:25.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2024-03-25T00:04:25.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:04:25.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:04:25.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:04:25.703 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:25.708+0000 7fe22c2937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:04:25.706 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:25.708+0000 7fe22c2937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:04:25.707 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:25.708+0000 7fe22c2937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:04:26.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:04:26.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:04:26.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:04:26.108 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:04:26.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:04:26.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:04:26.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:26.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:04:26.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:26.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:26.272 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:26.276+0000 7fe22c2937c0 -1 Falling back to public interface 2024-03-25T00:04:26.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:27.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:27.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:27.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:04:27.556 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:04:27.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:27.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:27.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:28.916 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:28.920+0000 7fe22c2937c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:04:28.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:28.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:28.971 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:04:28.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:04:28.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:28.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:29.658 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:30.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:30.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:30.660 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:04:30.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:04:30.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:30.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:31.082 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:32.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:32.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:32.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:04:32.084 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:04:32.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:32.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:32.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:33.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:33.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:33.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:04:33.551 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:04:33.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:33.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:34.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:35.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:35.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:35.172 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:04:35.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:04:35.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:35.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:35.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:36.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:36.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:36.611 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:04:36.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:04:36.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:36.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:37.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:38.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:38.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:38.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:04:38.162 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:04:38.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:38.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:38.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:39.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:39.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:39.615 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:04:39.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:04:39.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:39.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:40.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:41.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:41.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:41.154 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:04:41.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:04:41.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:41.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:41.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:42.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:42.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:42.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:04:42.609 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:04:42.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:42.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:43.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:44.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:44.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:44.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:04:44.862 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:04:44.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:44.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:45.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:46.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:46.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:46.728 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:04:46.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:04:46.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:46.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:47.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:48.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:48.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:48.221 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:04:48.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:04:48.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:48.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:48.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:49.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:49.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:49.753 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:04:49.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:04:49.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:49.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:50.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:51.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:51.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:51.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:04:51.588 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:04:51.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:51.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:52.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:53.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:53.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:53.068 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:04:53.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:04:53.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:53.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:53.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:04:54.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:04:54.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:04:54.916 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:04:54.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:04:54.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:04:54.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:04:55.358 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1988890948,v1:127.0.0.1:6803/1988890948] [v2:127.0.0.1:6804/1988890948,v1:127.0.0.1:6805/1988890948] exists,up 2d1be9c4-aa04-4dca-8a48-6c648474c1cf 2024-03-25T00:04:55.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:04:55.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:04:55.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:04:55.361 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-03-25T00:04:55.361 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:04:55.361 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:04:55.361 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:04:55.361 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:55.361 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:55.361 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-osd.0.asok 2024-03-25T00:04:55.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:04:55.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-osd.0.asok log flush 2024-03-25T00:04:55.457 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:04:55.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-03-25T00:04:55.472 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:04:16.508+0000 7f7d90210c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-03-25T00:04:55.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2024-03-25T00:04:55.475 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:04:26.276+0000 7fe22c2937c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-03-25T00:04:55.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-03-25T00:04:55.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:55.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:04:55.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:04:55.476 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:04:55.477 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:04:55.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:04:55.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:04:55.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:04:55.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:04:55.585 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:04:55.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:04:55.585 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:04:55.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:04:55.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:04:55.588 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:04:55.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:04:55.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:04:55.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:04:55.588 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:04:55.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:04:55.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:04:55.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:04:55.594 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:04:55.595 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:55.595 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:55.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:04:55.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:04:55.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:04:55.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:04:55.598 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:04:55.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:04:55.599 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:04:55.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:04:55.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:04:55.600 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:04:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:04:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:04:55.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:04:55.601 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:04:55.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:04:55.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:04:55.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:04:55.603 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:04:55.603 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:55.604 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:55.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:04:55.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:04:55.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:04:55.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:04:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:04:55.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-25T00:04:55.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2024-03-25T00:04:56.077 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:04:56.078 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:04:56.078 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:04:56.079 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:04:56.079 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:56.079 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:56.079 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:04:56.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2024-03-25T00:04:56.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:04:56.290 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:04:56.290 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:04:56.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-25T00:04:56.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:04:56.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:04:56.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:04:56.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:04:56.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:04:56.404 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:04:56.404 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:04:56.405 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:04:56.405 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:04:56.405 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:56.405 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:56.405 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:04:56.405 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:04:56.405 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-25T00:04:56.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-03-25T00:04:56.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:56.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:04:56.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:04:56.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:04:56.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-25T00:04:56.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:04:57.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:04:57.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:04:57.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:04:57.437 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:04:57.437 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:57.438 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:57.438 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:04:57.438 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:04:57.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:04:57.469 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-03-25T00:04:57.469 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:04:57.469 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:04:57.469 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:04:57.470 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:57.470 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:57.470 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:04:57.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:04:57.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-25T00:04:57.584 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:04:57.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2024-03-25T00:04:57.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-25T00:04:57.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:04:57.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:04:57.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:04:57.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-25T00:04:57.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-25T00:04:57.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:04:57.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:04:57.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:04:57.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-25T00:04:57.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-25T00:04:57.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:04:57.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:04:57.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-25T00:04:57.601 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:04:57.601 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:04:57.601 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:04:57.603 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:04:57.603 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:04:57.603 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:04:57.603 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:04:57.603 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:04:57.603 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:04:57.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:04:57.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:04:57.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-25T00:04:57.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-25T00:04:57.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:04:57.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:04:57.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:04:57.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:04:57.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:04:57.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2024-03-25T00:04:57.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-25T00:04:57.605 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:04:57.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=87022edd-efa1-4a31-af43-29c6e2f71e1e 2024-03-25T00:04:57.607 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 87022edd-efa1-4a31-af43-29c6e2f71e1e 2024-03-25T00:04:57.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 87022edd-efa1-4a31-af43-29c6e2f71e1e' 2024-03-25T00:04:57.607 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:04:57.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpvwBmmEtjJRAA5+oUqQI5TFaAbHMzBiQwMg== 2024-03-25T00:04:57.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpvwBmmEtjJRAA5+oUqQI5TFaAbHMzBiQwMg=="}' 2024-03-25T00:04:57.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 87022edd-efa1-4a31-af43-29c6e2f71e1e -i td/test-erasure-code-plugins/0/new.json 2024-03-25T00:04:57.926 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:04:57.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-25T00:04:57.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQCpvwBmmEtjJRAA5+oUqQI5TFaAbHMzBiQwMg== --osd-uuid 87022edd-efa1-4a31-af43-29c6e2f71e1e 2024-03-25T00:04:57.974 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:57.976+0000 7f9d3a7c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:04:58.003 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:58.008+0000 7f9d3a7c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:04:58.004 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:58.008+0000 7f9d3a7c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:04:58.005 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:58.008+0000 7f9d3a7c77c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:04:58.005 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:58.008+0000 7f9d3a7c77c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:04:58.005 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:58.008+0000 7f9d3a7c77c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:04:58.043 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:04:58.048+0000 7f9d3a7c77c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-25T00:05:04.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-25T00:05:04.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:05:04.590 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:05:04.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:05:04.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:05:05.136 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:05:05.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:05:05.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:05:05.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2024-03-25T00:05:05.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:05:05.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:05:05.167 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:05.171+0000 7f4150f237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:05:05.170 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:05.171+0000 7f4150f237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:05:05.170 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:05.171+0000 7f4150f237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:05:05.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:05:05.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:05:05.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:05:05.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:05:05.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:05:05.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:05.544 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:05:05.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:05:05.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:05.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:05.737 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:05.739+0000 7f4150f237c0 -1 Falling back to public interface 2024-03-25T00:05:05.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:06.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:06.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:06.972 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:05:06.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:05:06.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:06.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:07.374 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:08.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:08.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:08.376 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:05:08.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:05:08.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:08.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:08.413 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:08.415+0000 7f4150f237c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:05:08.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:09.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:09.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:09.965 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:05:09.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:05:09.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:09.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:10.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:11.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:11.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:11.425 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:05:11.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:05:11.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:11.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:11.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:12.896 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:12.896 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:12.896 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:05:12.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:05:12.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:12.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:13.419 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:14.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:14.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:14.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:05:14.422 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:05:14.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:14.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:14.848 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:15.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:15.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:15.850 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:05:15.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:05:15.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:15.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:16.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:17.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:17.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:17.323 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:05:17.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:05:17.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:17.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:17.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:18.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:18.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:18.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:05:18.768 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:05:18.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:18.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:19.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:20.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:20.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:20.329 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:05:20.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:05:20.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:20.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:20.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:21.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:21.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:21.831 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:05:21.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:05:21.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:21.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:22.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:23.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:23.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:23.647 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:05:23.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:05:23.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:23.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:24.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:25.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:25.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:25.383 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:05:25.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:05:25.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:25.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:26.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:27.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:27.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:27.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:05:27.521 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:05:27.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:27.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:28.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:29.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:29.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:29.355 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:05:29.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:05:29.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:29.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:30.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:31.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:31.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:31.589 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:05:31.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:05:31.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:31.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:32.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:33.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:33.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:33.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:05:33.452 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:05:33.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:33.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:34.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:35.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:35.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:35.073 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:05:35.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:05:35.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:35.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:35.477 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1601389953,v1:127.0.0.1:6803/1601389953] [v2:127.0.0.1:6804/1601389953,v1:127.0.0.1:6805/1601389953] exists,up 87022edd-efa1-4a31-af43-29c6e2f71e1e 2024-03-25T00:05:35.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:05:35.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:05:35.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:05:35.478 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-03-25T00:05:35.478 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:05:35.478 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:05:35.479 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:05:35.479 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:05:35.479 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:05:35.479 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-osd.0.asok 2024-03-25T00:05:35.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:05:35.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-osd.0.asok log flush 2024-03-25T00:05:35.574 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:05:35.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-03-25T00:05:35.589 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:04:56.120+0000 7fcdeb02cc80 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-03-25T00:05:35.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2024-03-25T00:05:35.590 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:05:05.743+0000 7f4150f237c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-03-25T00:05:35.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-03-25T00:05:35.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:05:35.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:05:35.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:05:35.592 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:05:35.592 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:05:35.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:05:35.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:05:35.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:05:35.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:05:35.700 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:05:35.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:05:35.702 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:05:35.703 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:05:35.703 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:05:35.703 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:05:35.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:05:35.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:05:35.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:05:35.705 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:05:35.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:05:35.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:05:35.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:05:35.712 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:05:35.712 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:05:35.712 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:05:35.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:05:35.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:05:35.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:05:35.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-03-25T00:05:35.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-03-25T00:05:35.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-25T00:05:35.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-25T00:05:35.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:05:35.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:05:35.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:05:35.718 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:05:35.719 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:05:35.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:05:35.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:05:35.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:05:35.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:05:35.720 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:05:35.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:05:35.720 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:05:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:05:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:05:35.721 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:05:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:05:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:05:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:05:35.721 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:05:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:05:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:05:35.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:05:35.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:05:35.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:05:35.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:05:35.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:05:35.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:05:35.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:05:35.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-25T00:05:35.724 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:05:35.725 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:05:35.725 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:05:35.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-25T00:05:35.727 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:05:35.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:05:35.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:05:35.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-25T00:05:35.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2024-03-25T00:05:35.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-25T00:05:35.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:05:35.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:05:35.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:05:35.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-25T00:05:35.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2024-03-25T00:05:36.228 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:05:36.229 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:05:36.229 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:05:36.229 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:05:36.229 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:05:36.229 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:05:36.229 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:05:36.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2024-03-25T00:05:36.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:05:36.455 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:05:36.456 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:05:36.456 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:05:36.456 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:05:36.456 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:05:36.457 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:05:36.457 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:05:36.457 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:05:36.458 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:05:36.458 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:05:36.458 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:05:36.458 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:05:36.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:05:36.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-25T00:05:36.574 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:05:36.574 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:05:36.574 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:05:36.574 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:05:36.575 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:05:36.575 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:05:36.575 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:05:36.575 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:05:36.575 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:05:36.575 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:05:36.576 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:05:36.576 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:05:36.576 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:05:36.576 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-25T00:05:36.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-03-25T00:05:36.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-25T00:05:36.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:05:36.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:05:36.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:05:36.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-25T00:05:36.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:05:37.599 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:05:37.599 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:05:37.599 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:05:37.599 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:05:37.599 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:05:37.600 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:05:37.600 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:05:37.600 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:05:37.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:05:37.663 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-03-25T00:05:37.664 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:05:37.664 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:05:37.664 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:05:37.664 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:05:37.665 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:05:37.665 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:05:37.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:05:37.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-25T00:05:37.767 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:05:37.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2024-03-25T00:05:37.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:05:37.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:05:37.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-25T00:05:37.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:05:37.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:05:37.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:05:37.783 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:05:37.783 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:05:37.783 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:05:37.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:05:37.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:05:37.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:05:37.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6f67ff0a-d31b-4adb-a7db-10263ad5bf33 2024-03-25T00:05:37.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6f67ff0a-d31b-4adb-a7db-10263ad5bf33' 2024-03-25T00:05:37.786 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 6f67ff0a-d31b-4adb-a7db-10263ad5bf33 2024-03-25T00:05:37.786 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:05:37.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRvwBmpPZMMBAAqaSo2DkTX2nSwlTZtqHuTA== 2024-03-25T00:05:37.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRvwBmpPZMMBAAqaSo2DkTX2nSwlTZtqHuTA=="}' 2024-03-25T00:05:37.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6f67ff0a-d31b-4adb-a7db-10263ad5bf33 -i td/test-erasure-code-plugins/0/new.json 2024-03-25T00:05:38.119 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:05:38.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-25T00:05:38.135 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQDRvwBmpPZMMBAAqaSo2DkTX2nSwlTZtqHuTA== --osd-uuid 6f67ff0a-d31b-4adb-a7db-10263ad5bf33 2024-03-25T00:05:38.163 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:38.167+0000 7f7320c907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:05:38.196 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:38.199+0000 7f7320c907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:05:38.197 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:38.199+0000 7f7320c907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:05:38.197 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:38.199+0000 7f7320c907c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:05:38.198 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:38.199+0000 7f7320c907c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:05:38.198 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:38.199+0000 7f7320c907c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:05:38.244 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:38.247+0000 7f7320c907c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-25T00:05:44.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-25T00:05:44.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:05:44.691 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:05:44.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:05:44.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:05:45.221 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:05:45.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:05:45.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2024-03-25T00:05:45.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:05:45.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:05:45.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:05:45.251 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:45.255+0000 7f67d7ccd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:05:45.253 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:45.255+0000 7f67d7ccd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:05:45.254 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:45.255+0000 7f67d7ccd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:05:45.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:05:45.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:05:45.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:05:45.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:05:45.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:05:45.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:45.655 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:05:45.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:05:45.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:45.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:45.807 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:45.811+0000 7f67d7ccd7c0 -1 Falling back to public interface 2024-03-25T00:05:46.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:47.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:47.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:47.077 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:05:47.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:05:47.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:47.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:47.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:48.512 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:05:48.515+0000 7f67d7ccd7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:05:48.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:48.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:48.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:05:48.538 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:05:48.539 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:48.539 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:49.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:50.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:50.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:50.249 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:05:50.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:05:50.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:50.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:50.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:51.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:51.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:51.763 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:05:51.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:05:51.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:51.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:52.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:53.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:53.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:53.183 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:05:53.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:05:53.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:53.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:53.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:54.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:54.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:54.733 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:05:54.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:05:54.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:54.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:55.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:56.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:56.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:56.161 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:05:56.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:05:56.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:56.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:56.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:57.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:57.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:57.728 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:05:57.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:05:57.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:57.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:58.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:05:59.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:05:59.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:05:59.174 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:05:59.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:05:59.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:05:59.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:05:59.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:00.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:00.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:00.782 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:06:00.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:06:00.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:00.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:01.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:02.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:02.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:02.205 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:06:02.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:06:02.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:02.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:02.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:03.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:03.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:03.974 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:06:03.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:06:03.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:03.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:04.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:05.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:05.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:05.910 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:06:05.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:06:05.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:05.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:07.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:08.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:08.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:08.030 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:06:08.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:06:08.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:08.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:09.260 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:10.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:10.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:10.262 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:06:10.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:06:10.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:10.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:10.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:11.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:11.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:11.962 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:06:11.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:06:11.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:11.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:12.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:13.641 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:13.641 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:13.642 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:06:13.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:06:13.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:13.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:14.082 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1549661397,v1:127.0.0.1:6803/1549661397] [v2:127.0.0.1:6804/1549661397,v1:127.0.0.1:6805/1549661397] exists,up 6f67ff0a-d31b-4adb-a7db-10263ad5bf33 2024-03-25T00:06:14.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:06:14.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:06:14.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:06:14.084 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-03-25T00:06:14.084 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:06:14.084 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:06:14.084 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:06:14.084 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:14.084 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:14.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-osd.0.asok 2024-03-25T00:06:14.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:06:14.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-osd.0.asok log flush 2024-03-25T00:06:14.194 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:06:14.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-03-25T00:06:14.211 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:05:36.267+0000 7f0e95918c80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-03-25T00:06:14.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2024-03-25T00:06:14.214 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:05:45.811+0000 7f67d7ccd7c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-03-25T00:06:14.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-03-25T00:06:14.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:06:14.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:06:14.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:06:14.216 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:06:14.216 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:06:14.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:06:14.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:06:14.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:06:14.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:06:14.526 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:06:14.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:06:14.528 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:06:14.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:06:14.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:06:14.530 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:06:14.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:06:14.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:06:14.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:06:14.533 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:06:14.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:06:14.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:06:14.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:06:14.543 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:06:14.543 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:14.543 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:14.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:06:14.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:06:14.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:06:14.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:06:14.549 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:06:14.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:06:14.551 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:06:14.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:06:14.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:06:14.554 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:06:14.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:06:14.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:06:14.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:06:14.554 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:06:14.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:06:14.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:06:14.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:06:14.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:06:14.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:14.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:14.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:06:14.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:06:14.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:06:14.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-03-25T00:06:14.561 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:06:14.562 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:14.562 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:14.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20241 2024-03-25T00:06:14.563 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:06:14.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:06:14.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:06:14.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-03-25T00:06:14.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2024-03-25T00:06:14.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-03-25T00:06:14.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:06:14.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:06:14.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:06:14.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-03-25T00:06:14.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2024-03-25T00:06:15.045 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:06:15.046 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:06:15.046 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:06:15.046 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:06:15.046 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:15.046 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:15.046 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:06:15.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2024-03-25T00:06:15.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:06:15.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:06:15.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:06:15.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:06:15.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:06:15.250 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:06:15.250 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:06:15.250 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:06:15.250 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:06:15.251 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:06:15.251 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:15.251 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:15.251 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:06:15.252 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:06:15.252 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get fsid 2024-03-25T00:06:15.371 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:06:15.371 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:06:15.371 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:06:15.371 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:06:15.371 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:06:15.372 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:06:15.372 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:06:15.372 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:06:15.372 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:06:15.372 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:15.372 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:15.372 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:06:15.373 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:06:15.373 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20241/ceph-mon.a.asok config get mon_host 2024-03-25T00:06:15.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-03-25T00:06:15.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-03-25T00:06:15.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:06:15.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:06:15.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:06:15.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-03-25T00:06:15.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:06:16.404 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:06:16.404 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:06:16.404 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:06:16.404 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:06:16.404 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:16.405 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:16.405 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:06:16.405 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:06:16.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:06:16.439 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-03-25T00:06:16.440 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:06:16.440 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:06:16.440 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:06:16.440 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:16.440 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:16.440 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-mon.a.asok 2024-03-25T00:06:16.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:06:16.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-mon.a.asok log flush 2024-03-25T00:06:16.538 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:06:16.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2024-03-25T00:06:16.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-03-25T00:06:16.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:06:16.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:06:16.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:06:16.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-03-25T00:06:16.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-03-25T00:06:16.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:06:16.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:06:16.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:06:16.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-03-25T00:06:16.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-03-25T00:06:16.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:06:16.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:06:16.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-03-25T00:06:16.555 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:06:16.555 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:06:16.555 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:06:16.555 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:06:16.555 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:16.555 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:16.559 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:06:16.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' 2024-03-25T00:06:16.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:06:16.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:06:16.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:06:16.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-03-25T00:06:16.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-03-25T00:06:16.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:06:16.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:06:16.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:06:16.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:06:16.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:06:16.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2024-03-25T00:06:16.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-03-25T00:06:16.560 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:06:16.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ab4f43fd-83a0-495f-b994-46bd8e0be44a 2024-03-25T00:06:16.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ab4f43fd-83a0-495f-b994-46bd8e0be44a' 2024-03-25T00:06:16.560 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 ab4f43fd-83a0-495f-b994-46bd8e0be44a 2024-03-25T00:06:16.561 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:06:16.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD4vwBmaHeTIhAAJc/kKN2lV7Ws942kMuHa1g== 2024-03-25T00:06:16.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD4vwBmaHeTIhAAJc/kKN2lV7Ws942kMuHa1g=="}' 2024-03-25T00:06:16.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ab4f43fd-83a0-495f-b994-46bd8e0be44a -i td/test-erasure-code-plugins/0/new.json 2024-03-25T00:06:16.877 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:06:16.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-03-25T00:06:16.893 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQD4vwBmaHeTIhAAJc/kKN2lV7Ws942kMuHa1g== --osd-uuid ab4f43fd-83a0-495f-b994-46bd8e0be44a 2024-03-25T00:06:16.925 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:16.926+0000 7fd68c2167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:06:16.954 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:16.958+0000 7fd68c2167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:06:16.955 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:16.958+0000 7fd68c2167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:06:16.957 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:16.958+0000 7fd68c2167c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:06:16.957 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:16.958+0000 7fd68c2167c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:06:16.957 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:16.958+0000 7fd68c2167c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-03-25T00:06:16.986 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:16.986+0000 7fd68c2167c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-03-25T00:06:23.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-03-25T00:06:23.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:06:23.204 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:06:23.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:06:23.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:06:23.763 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:06:23.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:06:23.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=185b99d8-b229-428d-bbdd-3360895c48d7 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20241/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2024-03-25T00:06:23.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:06:23.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:06:23.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:06:23.788 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:23.790+0000 7fdc4685a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:06:23.790 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:23.794+0000 7fdc4685a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:06:23.791 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:23.794+0000 7fdc4685a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:06:24.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:06:24.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:06:24.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:06:24.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:06:24.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:06:24.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:24.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:06:24.215 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:06:24.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:24.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:24.351 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:24.354+0000 7fdc4685a7c0 -1 Falling back to public interface 2024-03-25T00:06:24.628 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:25.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:25.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:25.630 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:06:25.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:06:25.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:25.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:26.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:27.036 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:27.038+0000 7fdc4685a7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:06:27.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:27.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:27.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:06:27.037 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:06:27.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:27.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:27.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:28.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:28.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:28.576 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:06:28.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:06:28.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:28.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:29.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:30.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:30.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:30.065 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:06:30.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:06:30.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:30.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:30.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:31.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:31.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:31.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:06:31.515 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:06:31.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:31.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:32.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:33.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:33.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:33.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:06:33.074 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:06:33.074 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:33.074 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:33.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:34.539 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:34.539 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:34.539 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:06:34.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:06:34.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:34.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:35.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:36.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:36.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:36.115 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:06:36.116 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:06:36.116 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:36.116 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:36.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:37.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:37.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:37.530 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:06:37.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:06:37.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:37.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:38.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:39.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:39.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:39.088 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:06:39.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:06:39.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:39.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:39.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:40.539 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:40.539 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:40.539 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:06:40.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:06:40.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:40.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:41.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:42.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:42.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:42.297 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:06:42.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:06:42.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:42.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:43.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:44.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:44.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:44.067 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:06:44.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:06:44.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:44.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:44.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:45.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:45.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:45.796 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:06:45.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:06:45.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:45.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:46.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:47.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:47.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:47.758 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:06:47.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:06:47.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:47.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:48.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:49.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:49.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:49.470 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:06:49.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:06:49.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:49.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:50.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:51.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:51.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:51.207 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:06:51.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:06:51.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:51.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:52.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:06:53.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:06:53.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:06:53.532 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:06:53.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:06:53.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:06:53.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:06:53.941 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/978767082,v1:127.0.0.1:6803/978767082] [v2:127.0.0.1:6804/978767082,v1:127.0.0.1:6805/978767082] exists,up ab4f43fd-83a0-495f-b994-46bd8e0be44a 2024-03-25T00:06:53.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:06:53.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:06:53.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:06:53.945 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-03-25T00:06:53.945 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:06:53.945 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:06:53.945 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:06:53.946 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:53.946 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:53.946 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20241/ceph-osd.0.asok 2024-03-25T00:06:53.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-03-25T00:06:53.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20241/ceph-osd.0.asok log flush 2024-03-25T00:06:54.051 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:06:54.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-03-25T00:06:54.067 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:06:15.086+0000 7eff7fa99c80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-03-25T00:06:54.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2024-03-25T00:06:54.069 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:06:24.354+0000 7fdc4685a7c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-03-25T00:06:54.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-03-25T00:06:54.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:06:54.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:06:54.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:06:54.071 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:06:54.071 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:06:54.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:06:54.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:06:54.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:06:54.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:06:54.382 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:06:54.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:06:54.383 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:06:54.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:06:54.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:06:54.386 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:06:54.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:06:54.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:06:54.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:06:54.388 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:06:54.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:06:54.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:06:54.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:06:54.397 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:06:54.398 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:54.398 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:54.398 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:06:54.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:06:54.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:06:54.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2024-03-25T00:06:54.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-25T00:06:54.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-code-plugins 0 2024-03-25T00:06:54.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-03-25T00:06:54.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-25T00:06:54.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-03-25T00:06:54.400 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:06:54.400 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:06:54.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:06:54.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:06:54.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:06:54.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:06:54.402 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:06:54.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:06:54.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:06:54.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:06:54.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:06:54.405 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:06:54.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:06:54.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:06:54.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:06:54.406 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:06:54.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:06:54.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-25T00:06:54.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-03-25T00:06:54.410 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:06:54.411 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:54.411 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20241 2024-03-25T00:06:54.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20241 2024-03-25T00:06:54.412 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:06:54.412 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:06:54.413 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-25T00:06:54.413 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-25T00:06:54.413 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-25T00:06:54.426 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2024-03-25T00:06:54.431 DEBUG:teuthology.orchestra.run.smithi080:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9c429fb6abfb5deed62697d56d8997d0f0d6d83f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2024-03-25T00:06:54.482 INFO:tasks.workunit.client.0.smithi080.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-25T00:06:54.490 INFO:tasks.workunit.client.0.smithi080.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-25T00:06:54.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T00:06:54.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T00:06:54.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-25T00:06:54.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-25T00:06:54.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-25T00:06:54.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-25T00:06:54.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-25T00:06:54.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-25T00:06:54.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-code 2024-03-25T00:06:54.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2024-03-25T00:06:54.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2024-03-25T00:06:54.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2024-03-25T00:06:54.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2024-03-25T00:06:54.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2024-03-25T00:06:54.493 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2024-03-25T00:06:54.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none ' 2024-03-25T00:06:54.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:06:54.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2024-03-25T00:06:54.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code 2024-03-25T00:06:54.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code 2024-03-25T00:06:54.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-03-25T00:06:54.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:06:54.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-03-25T00:06:54.495 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:06:54.495 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:06:54.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:06:54.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:06:54.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:06:54.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:06:54.500 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:06:54.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:06:54.500 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:06:54.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:06:54.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:06:54.501 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:06:54.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:06:54.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:06:54.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:06:54.502 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:06:54.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:06:54.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:06:54.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-03-25T00:06:54.502 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:06:54.503 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:54.503 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:06:54.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48948 2024-03-25T00:06:54.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:06:54.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:06:54.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code 2024-03-25T00:06:54.508 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:06:54.508 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:54.508 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:06:54.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.48948 2024-03-25T00:06:54.508 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:06:54.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-25T00:06:54.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-25T00:06:54.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:06:54.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2024-03-25T00:06:54.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2024-03-25T00:06:54.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code 2024-03-25T00:06:54.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:06:54.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:06:54.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:06:54.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code/a 2024-03-25T00:06:54.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2024-03-25T00:06:54.948 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:06:54.949 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:06:54.949 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:06:54.949 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:06:54.949 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:54.950 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:06:54.950 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:06:54.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T00:06:55.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:06:55.159 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:06:55.159 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:06:55.159 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:06:55.160 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:06:55.161 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:06:55.161 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:06:55.161 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:06:55.161 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:06:55.161 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:06:55.161 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:55.161 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:06:55.161 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48948/ceph-mon.a.asok 2024-03-25T00:06:55.161 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:06:55.161 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48948/ceph-mon.a.asok config get fsid 2024-03-25T00:06:55.277 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:06:55.277 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:06:55.277 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:06:55.277 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:06:55.278 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:06:55.278 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:06:55.278 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:06:55.278 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:06:55.278 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:06:55.279 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:55.279 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:06:55.279 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48948/ceph-mon.a.asok 2024-03-25T00:06:55.279 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:06:55.279 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48948/ceph-mon.a.asok config get mon_host 2024-03-25T00:06:55.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2024-03-25T00:06:55.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code 2024-03-25T00:06:55.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:06:55.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:06:55.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:06:55.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code/x 2024-03-25T00:06:55.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:06:56.298 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:06:56.298 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:06:56.298 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:06:56.299 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:06:56.299 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:56.299 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:06:56.299 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:06:56.299 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:06:56.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:06:56.335 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2024-03-25T00:06:56.335 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:06:56.335 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:06:56.335 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:06:56.335 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:56.335 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:06:56.335 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48948/ceph-mon.a.asok 2024-03-25T00:06:56.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2024-03-25T00:06:56.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.48948/ceph-mon.a.asok log flush 2024-03-25T00:06:56.434 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:06:56.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2024-03-25T00:06:56.450 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:06:54.994+0000 7f5125f6dc80 0 load: jerasure load: lrc 2024-03-25T00:06:56.451 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2024-03-25T00:06:56.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-03-25T00:06:56.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2024-03-25T00:06:56.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-03-25T00:06:56.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:06:56.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:06:56.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:06:56.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/0 2024-03-25T00:06:56.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:06:56.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:06:56.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:06:56.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:06:56.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2024-03-25T00:06:56.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2024-03-25T00:06:56.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:06:56.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:06:56.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:06:56.456 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:06:56.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:06:56.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:06:56.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:06:56.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:06:56.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:06:56.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:06:56.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/0 2024-03-25T00:06:56.459 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:06:56.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c294b42-3a0f-45fc-ba64-8079575ef12d 2024-03-25T00:06:56.460 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 9c294b42-3a0f-45fc-ba64-8079575ef12d 2024-03-25T00:06:56.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9c294b42-3a0f-45fc-ba64-8079575ef12d' 2024-03-25T00:06:56.460 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:06:56.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAgwABmnKaaHBAAGUyJ1lpFLSopvWGU1wmkCQ== 2024-03-25T00:06:56.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAgwABmnKaaHBAAGUyJ1lpFLSopvWGU1wmkCQ=="}' 2024-03-25T00:06:56.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c294b42-3a0f-45fc-ba64-8079575ef12d -i td/test-erasure-code/0/new.json 2024-03-25T00:06:56.789 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:06:56.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/0/new.json 2024-03-25T00:06:56.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgwABmnKaaHBAAGUyJ1lpFLSopvWGU1wmkCQ== --osd-uuid 9c294b42-3a0f-45fc-ba64-8079575ef12d 2024-03-25T00:06:56.835 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:56.838+0000 7fbb221da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:06:56.866 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:56.870+0000 7fbb221da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:06:56.867 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:56.870+0000 7fbb221da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:06:56.868 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:56.870+0000 7fbb221da7c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-03-25T00:06:56.868 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:56.870+0000 7fbb221da7c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-03-25T00:06:56.868 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:56.870+0000 7fbb221da7c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-03-25T00:06:56.898 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:06:56.902+0000 7fbb221da7c0 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2024-03-25T00:07:03.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/0/keyring 2024-03-25T00:07:03.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:07:03.441 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:07:03.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:07:03.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:07:03.996 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:07:03.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:07:03.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:07:03.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:07:03.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:07:03.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:07:04.026 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:04.030+0000 7f0a86baa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:07:04.029 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:04.030+0000 7f0a86baa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:07:04.030 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:04.034+0000 7f0a86baa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:07:04.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:07:04.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:07:04.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:07:04.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:07:04.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:07:04.380 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:07:04.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:04.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:07:04.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:04.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:04.595 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:04.598+0000 7f0a86baa7c0 -1 Falling back to public interface 2024-03-25T00:07:04.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:05.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:05.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:05.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:07:05.811 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:07:05.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:05.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:06.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:07.174 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:07.178+0000 7f0a86baa7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:07:07.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:07.215 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:07:07.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:07.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:07:07.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:07.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:07.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:08.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:08.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:08.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:07:08.932 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:07:08.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:08.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:09.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:10.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:10.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:10.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:07:10.408 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:07:10.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:10.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:10.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:11.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:11.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:11.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:07:11.853 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:07:11.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:11.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:12.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:13.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:13.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:13.430 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:07:13.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:07:13.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:13.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:13.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:14.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:14.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:14.854 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:07:14.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:07:14.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:14.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:15.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:16.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:16.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:16.326 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:07:16.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:07:16.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:16.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:16.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:17.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:17.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:17.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:07:17.799 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:07:17.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:17.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:18.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:19.315 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:19.315 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:19.315 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:07:19.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:07:19.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:19.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:19.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:20.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:20.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:20.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:07:20.734 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:07:20.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:20.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:22.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:23.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:23.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:23.197 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:07:23.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:07:23.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:23.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:23.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:24.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:25.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:25.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:07:25.000 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:07:25.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:25.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:25.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:26.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:26.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:26.396 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:07:26.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:07:26.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:26.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:27.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:28.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:28.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:28.194 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:07:28.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:07:28.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:28.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:28.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:29.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:29.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:29.824 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:07:29.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:07:29.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:29.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:30.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:31.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:31.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:31.473 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:07:31.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:07:31.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:31.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:32.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:33.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:33.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:33.407 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:07:33.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:07:33.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:33.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:07:33.860 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2061753732,v1:127.0.0.1:6803/2061753732] [v2:127.0.0.1:6804/2061753732,v1:127.0.0.1:6805/2061753732] exists,up 9c294b42-3a0f-45fc-ba64-8079575ef12d 2024-03-25T00:07:33.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:07:33.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:07:33.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:07:33.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-03-25T00:07:33.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2024-03-25T00:07:33.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/1 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:07:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:07:33.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:07:33.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:07:33.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:07:33.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:07:33.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:07:33.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:07:33.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:07:33.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/1 2024-03-25T00:07:33.867 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:07:33.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=36491923-a70e-4a0f-b812-1b33226390b4 2024-03-25T00:07:33.868 INFO:tasks.workunit.client.0.smithi080.stdout:add osd1 36491923-a70e-4a0f-b812-1b33226390b4 2024-03-25T00:07:33.868 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 36491923-a70e-4a0f-b812-1b33226390b4' 2024-03-25T00:07:33.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:07:33.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBFwABmpnhGNRAACu8LqWFkvBVQgkd3OKMjaA== 2024-03-25T00:07:33.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBFwABmpnhGNRAACu8LqWFkvBVQgkd3OKMjaA=="}' 2024-03-25T00:07:33.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 36491923-a70e-4a0f-b812-1b33226390b4 -i td/test-erasure-code/1/new.json 2024-03-25T00:07:34.370 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:07:34.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/1/new.json 2024-03-25T00:07:34.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFwABmpnhGNRAACu8LqWFkvBVQgkd3OKMjaA== --osd-uuid 36491923-a70e-4a0f-b812-1b33226390b4 2024-03-25T00:07:34.421 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:34.421+0000 7f5bd43af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:07:34.464 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:34.465+0000 7f5bd43af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:07:34.466 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:34.469+0000 7f5bd43af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:07:34.467 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:34.469+0000 7f5bd43af7c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-03-25T00:07:34.467 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:34.469+0000 7f5bd43af7c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-03-25T00:07:34.467 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:34.469+0000 7f5bd43af7c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-03-25T00:07:34.512 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:34.513+0000 7f5bd43af7c0 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2024-03-25T00:07:41.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/1/keyring 2024-03-25T00:07:41.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:07:41.249 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd1 key to auth repository 2024-03-25T00:07:41.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T00:07:41.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:07:41.803 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.1 2024-03-25T00:07:41.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T00:07:41.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:07:41.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:07:41.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:07:41.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:07:41.834 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:41.837+0000 7f6b491c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:07:41.836 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:41.837+0000 7f6b491c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:07:41.837 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:41.841+0000 7f6b491c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:07:42.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T00:07:42.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:07:42.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:07:42.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:07:42.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:07:42.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:42.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:07:42.206 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:07:42.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:42.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:42.391 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:42.393+0000 7f6b491c27c0 -1 Falling back to public interface 2024-03-25T00:07:42.628 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:43.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:43.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:43.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:07:43.631 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:07:43.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:43.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:44.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:45.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:45.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:45.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:07:45.036 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:07:45.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:45.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:45.053 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:07:45.057+0000 7f6b491c27c0 -1 osd.1 0 log_to_monitors true 2024-03-25T00:07:45.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:46.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:46.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:46.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:07:46.473 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:07:46.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:46.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:47.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:48.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:48.139 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:07:48.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:48.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:07:48.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:48.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:48.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:49.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:49.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:49.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:07:49.575 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:07:49.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:49.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:50.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:51.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:51.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:51.133 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:07:51.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:07:51.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:51.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:52.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:52.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:52.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:07:52.708 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:07:52.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:52.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:53.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:54.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:54.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:54.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:07:54.262 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:07:54.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:54.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:54.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:55.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:55.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:55.693 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:07:55.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:07:55.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:55.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:56.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:57.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:57.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:57.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:07:57.186 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:07:57.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:57.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:57.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:07:58.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:07:58.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:07:58.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:07:58.753 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:07:58.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:07:58.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:07:59.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:00.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:00.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:00.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:08:00.393 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:08:00.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:00.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:08:01.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:02.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:02.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:02.089 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:08:02.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:08:02.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:02.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:08:03.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:04.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:04.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:04.219 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:08:04.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:08:04.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:04.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:08:05.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:06.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:06.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:06.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:08:06.036 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:08:06.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:06.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:08:06.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:07.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:07.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:07.783 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:08:07.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:08:07.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:07.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:08:08.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:09.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:09.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:09.911 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:08:09.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:08:09.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:09.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:08:10.868 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:11.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:11.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:11.870 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:08:11.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:08:11.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:11.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:08:12.376 INFO:tasks.workunit.client.0.smithi080.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4118018297,v1:127.0.0.1:6811/4118018297] [v2:127.0.0.1:6812/4118018297,v1:127.0.0.1:6813/4118018297] exists,up 36491923-a70e-4a0f-b812-1b33226390b4 2024-03-25T00:08:12.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:08:12.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/2 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:08:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:08:12.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2024-03-25T00:08:12.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2024-03-25T00:08:12.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:08:12.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:08:12.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:08:12.386 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:08:12.386 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:08:12.386 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:08:12.386 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:08:12.386 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:08:12.386 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:08:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:08:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/2 2024-03-25T00:08:12.388 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:08:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=596684b5-ed84-4672-82de-3c08f8de8e2a 2024-03-25T00:08:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 596684b5-ed84-4672-82de-3c08f8de8e2a' 2024-03-25T00:08:12.388 INFO:tasks.workunit.client.0.smithi080.stdout:add osd2 596684b5-ed84-4672-82de-3c08f8de8e2a 2024-03-25T00:08:12.388 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:08:12.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBswABmWmRnGBAALxA/kkZzbpstRyPKGI3TRA== 2024-03-25T00:08:12.403 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBswABmWmRnGBAALxA/kkZzbpstRyPKGI3TRA=="}' 2024-03-25T00:08:12.403 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 596684b5-ed84-4672-82de-3c08f8de8e2a -i td/test-erasure-code/2/new.json 2024-03-25T00:08:12.925 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:08:12.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/2/new.json 2024-03-25T00:08:12.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBswABmWmRnGBAALxA/kkZzbpstRyPKGI3TRA== --osd-uuid 596684b5-ed84-4672-82de-3c08f8de8e2a 2024-03-25T00:08:12.969 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:12.969+0000 7f11198907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:13.031 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:13.033+0000 7f11198907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:13.033 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:13.033+0000 7f11198907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:13.034 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:13.037+0000 7f11198907c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-03-25T00:08:13.034 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:13.037+0000 7f11198907c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-03-25T00:08:13.034 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:13.037+0000 7f11198907c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-03-25T00:08:13.155 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:13.157+0000 7f11198907c0 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2024-03-25T00:08:20.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/2/keyring 2024-03-25T00:08:20.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:08:20.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T00:08:20.177 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd2 key to auth repository 2024-03-25T00:08:20.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:08:20.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T00:08:20.729 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:08:20.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:08:20.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:08:20.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:08:20.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:08:20.761 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:20.765+0000 7f929d9307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:20.763 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:20.765+0000 7f929d9307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:20.765 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:20.769+0000 7f929d9307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:21.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T00:08:21.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:08:21.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:08:21.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:08:21.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:08:21.193 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:08:21.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:21.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:08:21.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:21.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:21.302 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:21.305+0000 7f929d9307c0 -1 Falling back to public interface 2024-03-25T00:08:21.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:22.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:22.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:22.631 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:08:22.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:08:22.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:22.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:23.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:23.971 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:23.973+0000 7f929d9307c0 -1 osd.2 0 log_to_monitors true 2024-03-25T00:08:24.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:24.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:24.049 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:08:24.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:08:24.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:24.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:24.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:25.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:25.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:25.759 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:08:25.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:08:25.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:25.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:26.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:27.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:27.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:27.230 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:08:27.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:08:27.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:27.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:27.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:28.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:28.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:28.726 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:08:28.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:08:28.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:28.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:29.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:30.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:30.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:30.191 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:08:30.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:08:30.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:30.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:30.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:31.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:31.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:31.662 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:08:31.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:08:31.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:31.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:32.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:33.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:33.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:33.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:08:33.269 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:08:33.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:33.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:33.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:34.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:34.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:34.730 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:08:34.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:08:34.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:34.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:35.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:36.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:36.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:36.214 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:08:36.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:08:36.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:36.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:36.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:37.668 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:37.668 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:37.669 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:08:37.669 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:08:37.669 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:37.669 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:38.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:39.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:39.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:39.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:08:39.166 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:08:39.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:39.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:39.848 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:40.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:40.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:40.850 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:08:40.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:08:40.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:40.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:41.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:42.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:42.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:42.715 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:08:42.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:08:42.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:42.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:43.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:44.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:44.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:44.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:08:44.697 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:08:44.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:44.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:45.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:46.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:46.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:46.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:08:46.624 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:08:46.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:46.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:47.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:48.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:48.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:48.672 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:08:48.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:08:48.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:48.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:49.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:08:50.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:08:50.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:50.554 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:08:50.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:08:50.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:50.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:08:51.097 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3360553850,v1:127.0.0.1:6819/3360553850] [v2:127.0.0.1:6820/3360553850,v1:127.0.0.1:6821/3360553850] exists,up 596684b5-ed84-4672-82de-3c08f8de8e2a 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/3 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:08:51.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:08:51.101 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:08:51.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/3 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:08:51.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a585fe77-b984-4429-9348-5aa201863183 2024-03-25T00:08:51.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 a585fe77-b984-4429-9348-5aa201863183' 2024-03-25T00:08:51.105 INFO:tasks.workunit.client.0.smithi080.stdout:add osd3 a585fe77-b984-4429-9348-5aa201863183 2024-03-25T00:08:51.105 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:08:51.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTwABmaiHeBxAA0V4XOi/XIkLAnvb/jO67/w== 2024-03-25T00:08:51.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTwABmaiHeBxAA0V4XOi/XIkLAnvb/jO67/w=="}' 2024-03-25T00:08:51.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a585fe77-b984-4429-9348-5aa201863183 -i td/test-erasure-code/3/new.json 2024-03-25T00:08:51.674 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:08:51.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/3/new.json 2024-03-25T00:08:51.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTwABmaiHeBxAA0V4XOi/XIkLAnvb/jO67/w== --osd-uuid a585fe77-b984-4429-9348-5aa201863183 2024-03-25T00:08:51.724 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:51.728+0000 7f2dba3507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:51.727 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:51.732+0000 7f2dba3507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:51.728 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:51.736+0000 7f2dba3507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:51.729 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:51.736+0000 7f2dba3507c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-03-25T00:08:51.729 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:51.736+0000 7f2dba3507c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-03-25T00:08:51.729 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:51.736+0000 7f2dba3507c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-03-25T00:08:51.767 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:51.772+0000 7f2dba3507c0 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2024-03-25T00:08:58.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/3/keyring 2024-03-25T00:08:58.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:08:58.539 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd3 key to auth repository 2024-03-25T00:08:58.539 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T00:08:58.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:08:59.120 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.3 2024-03-25T00:08:59.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T00:08:59.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:08:59.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:08:59.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:08:59.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:08:59.152 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:59.156+0000 7efde63c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:59.154 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:59.160+0000 7efde63c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:59.155 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:59.160+0000 7efde63c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:08:59.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T00:08:59.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:08:59.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:08:59.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:08:59.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:08:59.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:08:59.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:08:59.572 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:08:59.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:08:59.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:08:59.738 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:08:59.744+0000 7efde63c57c0 -1 Falling back to public interface 2024-03-25T00:08:59.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:00.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:00.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:00.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:09:00.958 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:09:00.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:00.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:01.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:09:02.373 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:09:02.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:02.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:02.394 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:02.400+0000 7efde63c57c0 -1 osd.3 0 log_to_monitors true 2024-03-25T00:09:02.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:03.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:03.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:03.964 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:09:03.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:09:03.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:03.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:04.465 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:05.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:05.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:05.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:09:05.468 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:09:05.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:05.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:06.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:07.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:07.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:07.054 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:09:07.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:09:07.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:07.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:07.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:08.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:08.715 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:09:08.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:08.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:09:08.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:08.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:09.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:10.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:10.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:10.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:09:10.167 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:09:10.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:10.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:10.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:11.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:11.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:11.576 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:09:11.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:09:11.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:11.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:12.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:13.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:13.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:13.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:09:13.122 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:09:13.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:13.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:13.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:14.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:14.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:14.739 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:09:14.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:09:14.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:14.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:15.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:16.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:16.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:16.340 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:09:16.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:09:16.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:16.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:16.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:17.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:17.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:17.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:09:17.790 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:09:17.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:17.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:18.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:19.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:19.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:19.494 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:09:19.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:09:19.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:19.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:20.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:21.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:21.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:21.711 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:09:21.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:09:21.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:21.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:22.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:23.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:23.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:23.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:09:23.728 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:09:23.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:23.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:24.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:25.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:25.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:25.334 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:09:25.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:09:25.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:25.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:26.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:27.481 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:09:27.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:09:27.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:27.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:28.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:29.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:29.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:29.456 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:09:29.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:09:29.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:29.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:09:29.872 INFO:tasks.workunit.client.0.smithi080.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/119460608,v1:127.0.0.1:6827/119460608] [v2:127.0.0.1:6828/119460608,v1:127.0.0.1:6829/119460608] exists,up a585fe77-b984-4429-9348-5aa201863183 2024-03-25T00:09:29.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:09:29.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:09:29.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:09:29.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/4 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:09:29.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:09:29.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:09:29.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:09:29.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:09:29.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:09:29.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:09:29.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:09:29.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:09:29.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/4 2024-03-25T00:09:29.879 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:09:29.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2eaea375-97fe-457e-9d9c-54e15e989838 2024-03-25T00:09:29.880 INFO:tasks.workunit.client.0.smithi080.stdout:add osd4 2eaea375-97fe-457e-9d9c-54e15e989838 2024-03-25T00:09:29.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 2eaea375-97fe-457e-9d9c-54e15e989838' 2024-03-25T00:09:29.880 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:09:29.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5wABmRQMFNhAAbywaM08XS4cQmeBktMt4Pg== 2024-03-25T00:09:29.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5wABmRQMFNhAAbywaM08XS4cQmeBktMt4Pg=="}' 2024-03-25T00:09:29.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2eaea375-97fe-457e-9d9c-54e15e989838 -i td/test-erasure-code/4/new.json 2024-03-25T00:09:30.380 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:09:30.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/4/new.json 2024-03-25T00:09:30.398 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5wABmRQMFNhAAbywaM08XS4cQmeBktMt4Pg== --osd-uuid 2eaea375-97fe-457e-9d9c-54e15e989838 2024-03-25T00:09:30.431 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:30.436+0000 7f613b65b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:09:30.457 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:30.464+0000 7f613b65b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:09:30.458 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:30.464+0000 7f613b65b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:09:30.459 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:30.464+0000 7f613b65b7c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-03-25T00:09:30.459 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:30.464+0000 7f613b65b7c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-03-25T00:09:30.459 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:30.464+0000 7f613b65b7c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-03-25T00:09:30.496 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:30.504+0000 7f613b65b7c0 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2024-03-25T00:09:37.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/4/keyring 2024-03-25T00:09:37.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:09:37.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-25T00:09:37.780 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd4 key to auth repository 2024-03-25T00:09:37.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:09:38.316 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.4 2024-03-25T00:09:38.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-25T00:09:38.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:09:38.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:09:38.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:09:38.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:09:38.347 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:38.351+0000 7f42469257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:09:38.349 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:38.355+0000 7f42469257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:09:38.350 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:38.355+0000 7f42469257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:09:38.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-25T00:09:38.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:09:38.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-25T00:09:38.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:09:38.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:09:38.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:38.707 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:09:38.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:09:38.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:38.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:38.906 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:38.911+0000 7f42469257c0 -1 Falling back to public interface 2024-03-25T00:09:39.131 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:40.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:40.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:40.133 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:09:40.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:09:40.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:40.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:40.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:41.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:41.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:41.536 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:09:41.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:09:41.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:41.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:41.598 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:09:41.603+0000 7f42469257c0 -1 osd.4 0 log_to_monitors true 2024-03-25T00:09:41.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:42.977 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:42.977 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:42.977 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:09:42.978 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:09:42.978 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:42.978 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:43.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:44.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:44.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:44.440 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:09:44.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:09:44.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:44.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:45.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:46.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:46.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:46.106 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:09:46.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:09:46.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:46.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:46.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:47.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:47.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:47.530 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:09:47.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:09:47.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:47.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:48.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:49.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:49.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:49.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:09:49.050 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:09:49.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:49.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:49.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:50.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:50.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:50.682 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:09:50.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:09:50.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:50.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:51.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:52.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:52.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:52.146 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:09:52.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:09:52.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:52.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:52.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:53.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:53.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:53.738 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:09:53.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:09:53.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:53.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:54.350 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:55.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:55.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:55.352 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:09:55.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:09:55.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:55.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:55.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:56.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:56.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:56.836 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:09:56.836 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:09:56.836 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:56.836 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:57.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:09:58.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:09:58.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:09:58.382 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:09:58.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:09:58.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:09:58.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:09:59.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:00.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:00.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:00.031 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:10:00.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:10:00.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:00.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:10:00.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:01.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:01.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:01.726 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:10:01.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:10:01.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:01.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:10:02.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:03.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:03.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:03.389 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:10:03.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:10:03.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:03.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:10:04.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:05.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:05.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:05.308 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:10:05.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:10:05.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:05.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:10:06.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:07.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:07.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:07.255 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:10:07.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:10:07.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:07.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:10:08.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:09.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:09.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:09.129 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-25T00:10:09.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T00:10:09.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:09.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:10:09.540 INFO:tasks.workunit.client.0.smithi080.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4209744420,v1:127.0.0.1:6835/4209744420] [v2:127.0.0.1:6836/4209744420,v1:127.0.0.1:6837/4209744420] exists,up 2eaea375-97fe-457e-9d9c-54e15e989838 2024-03-25T00:10:09.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:10:09.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:10:09.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:10:09.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-03-25T00:10:09.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2024-03-25T00:10:09.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-03-25T00:10:09.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:10:09.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-25T00:10:09.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:10:09.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/5 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:10:09.544 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:10:09.545 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:10:09.546 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:10:09.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/5 2024-03-25T00:10:09.548 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:10:09.548 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3ea4677-f7e5-4cb3-982a-e71a90fb9766 2024-03-25T00:10:09.548 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 f3ea4677-f7e5-4cb3-982a-e71a90fb9766' 2024-03-25T00:10:09.548 INFO:tasks.workunit.client.0.smithi080.stdout:add osd5 f3ea4677-f7e5-4cb3-982a-e71a90fb9766 2024-03-25T00:10:09.548 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:10:09.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhwABm0t8LIhAAebzFMa7WVoKMmSLo6S3Smw== 2024-03-25T00:10:09.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhwABm0t8LIhAAebzFMa7WVoKMmSLo6S3Smw=="}' 2024-03-25T00:10:09.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3ea4677-f7e5-4cb3-982a-e71a90fb9766 -i td/test-erasure-code/5/new.json 2024-03-25T00:10:10.057 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:10:10.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/5/new.json 2024-03-25T00:10:10.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhwABm0t8LIhAAebzFMa7WVoKMmSLo6S3Smw== --osd-uuid f3ea4677-f7e5-4cb3-982a-e71a90fb9766 2024-03-25T00:10:10.105 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:10.111+0000 7ff7144b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:10.152 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:10.159+0000 7ff7144b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:10.154 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:10.159+0000 7ff7144b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:10.154 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:10.159+0000 7ff7144b17c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-03-25T00:10:10.154 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:10.159+0000 7ff7144b17c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-03-25T00:10:10.155 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:10.159+0000 7ff7144b17c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-03-25T00:10:10.225 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:10.231+0000 7ff7144b17c0 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2024-03-25T00:10:17.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/5/keyring 2024-03-25T00:10:17.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:10:17.146 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd5 key to auth repository 2024-03-25T00:10:17.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-25T00:10:17.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:10:17.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-25T00:10:17.742 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.5 2024-03-25T00:10:17.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:10:17.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:10:17.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:10:17.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:10:17.773 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:17.779+0000 7f83336207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:17.789 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:17.795+0000 7f83336207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:17.790 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:17.795+0000 7f83336207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:18.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-25T00:10:18.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:10:18.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T00:10:18.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:10:18.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:10:18.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:18.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:10:18.152 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:10:18.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:18.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:18.365 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:18.371+0000 7f83336207c0 -1 Falling back to public interface 2024-03-25T00:10:18.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:19.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:19.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:19.640 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:10:19.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:10:19.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:19.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:20.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:21.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:21.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:21.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:10:21.060 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:10:21.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:21.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:21.125 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:21.131+0000 7f83336207c0 -1 osd.5 0 log_to_monitors true 2024-03-25T00:10:21.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:22.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:22.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:22.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:10:22.475 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:10:22.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:22.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:23.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:24.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:24.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:24.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:10:24.107 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:10:24.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:24.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:24.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:25.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:25.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:25.649 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:10:25.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:10:25.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:25.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:26.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:27.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:27.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:27.146 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:10:27.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:10:27.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:27.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:27.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:28.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:28.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:28.769 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:10:28.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:10:28.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:28.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:29.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:30.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:30.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:30.451 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:10:30.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:10:30.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:30.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:30.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:31.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:31.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:31.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:10:31.852 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:10:31.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:31.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:32.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:33.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:33.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:33.444 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:10:33.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:10:33.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:33.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:33.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:34.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:34.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:34.872 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:10:34.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:10:34.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:34.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:35.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:36.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:36.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:36.494 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:10:36.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:10:36.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:36.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:37.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:38.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:38.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:38.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:10:38.128 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:10:38.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:38.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:39.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:40.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:40.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:40.020 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:10:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:10:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:40.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:41.770 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:41.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:41.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:10:41.771 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:10:41.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:41.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:42.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:43.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:43.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:43.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:10:43.698 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:10:43.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:43.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:44.893 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:45.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:45.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:45.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:10:45.895 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:10:45.896 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:45.896 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:46.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:47.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:47.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:47.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:10:47.815 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:10:47.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:47.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:48.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:49.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:49.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:49.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T00:10:49.308 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-25T00:10:49.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:49.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:10:49.732 INFO:tasks.workunit.client.0.smithi080.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2652336594,v1:127.0.0.1:6843/2652336594] [v2:127.0.0.1:6844/2652336594,v1:127.0.0.1:6845/2652336594] exists,up f3ea4677-f7e5-4cb3-982a-e71a90fb9766 2024-03-25T00:10:49.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:10:49.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:10:49.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:10:49.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-03-25T00:10:49.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2024-03-25T00:10:49.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-03-25T00:10:49.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:10:49.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-25T00:10:49.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:10:49.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/6 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:10:49.736 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:10:49.737 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:10:49.737 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:10:49.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:10:49.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:10:49.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:10:49.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:10:49.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:10:49.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:10:49.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:10:49.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:10:49.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:10:49.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:10:49.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:10:49.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:10:49.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/6 2024-03-25T00:10:49.739 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:10:49.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8b68bba8-b275-41e9-9674-28c4f1e0a905 2024-03-25T00:10:49.739 INFO:tasks.workunit.client.0.smithi080.stdout:add osd6 8b68bba8-b275-41e9-9674-28c4f1e0a905 2024-03-25T00:10:49.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 8b68bba8-b275-41e9-9674-28c4f1e0a905' 2024-03-25T00:10:49.740 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:10:49.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJwQBmRWnTLRAAtnQS6LJb0SjFYVsTiiYpTg== 2024-03-25T00:10:49.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJwQBmRWnTLRAAtnQS6LJb0SjFYVsTiiYpTg=="}' 2024-03-25T00:10:49.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8b68bba8-b275-41e9-9674-28c4f1e0a905 -i td/test-erasure-code/6/new.json 2024-03-25T00:10:50.233 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:10:50.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/6/new.json 2024-03-25T00:10:50.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJwQBmRWnTLRAAtnQS6LJb0SjFYVsTiiYpTg== --osd-uuid 8b68bba8-b275-41e9-9674-28c4f1e0a905 2024-03-25T00:10:50.283 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:50.286+0000 7fa9d56767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:50.327 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:50.330+0000 7fa9d56767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:50.329 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:50.334+0000 7fa9d56767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:50.329 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:50.334+0000 7fa9d56767c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-03-25T00:10:50.330 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:50.334+0000 7fa9d56767c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-03-25T00:10:50.330 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:50.334+0000 7fa9d56767c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-03-25T00:10:50.451 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:50.454+0000 7fa9d56767c0 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2024-03-25T00:10:57.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/6/keyring 2024-03-25T00:10:57.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:10:57.042 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd6 key to auth repository 2024-03-25T00:10:57.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-25T00:10:57.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:10:57.595 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.6 2024-03-25T00:10:57.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-25T00:10:57.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:10:57.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:10:57.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:10:57.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:10:57.626 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:57.630+0000 7f6a378087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:57.628 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:57.634+0000 7f6a378087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:57.629 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:57.634+0000 7f6a378087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:10:57.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-25T00:10:57.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:10:57.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-25T00:10:57.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:10:57.981 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:10:57.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:10:57.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:57.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:10:57.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:57.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:10:58.181 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:10:58.186+0000 7f6a378087c0 -1 Falling back to public interface 2024-03-25T00:10:58.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:10:59.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:10:59.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:10:59.441 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:10:59.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:10:59.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:10:59.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:10:59.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:00.771 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:00.774+0000 7f6a378087c0 -1 osd.6 0 log_to_monitors true 2024-03-25T00:11:00.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:00.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:00.840 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:11:00.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:11:00.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:00.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:01.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:02.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:02.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:02.435 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:11:02.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:11:02.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:02.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:02.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:03.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:03.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:03.862 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:11:03.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:11:03.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:03.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:04.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:05.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:05.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:05.330 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:11:05.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:11:05.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:05.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:05.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:06.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:06.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:06.901 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:11:06.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:11:06.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:06.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:07.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:08.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:08.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:08.324 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:11:08.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:11:08.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:08.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:08.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:09.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:09.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:09.763 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:11:09.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:11:09.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:09.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:10.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:11.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:11.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:11.216 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:11:11.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:11:11.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:11.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:11.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:12.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:12.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:12.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:11:12.721 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:11:12.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:12.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:13.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:14.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:14.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:14.164 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:11:14.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:11:14.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:14.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:14.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:15.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:15.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:15.755 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:11:15.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:11:15.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:15.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:16.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:17.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:17.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:17.622 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:11:17.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:11:17.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:17.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:18.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:19.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:19.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:19.879 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:11:19.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:11:19.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:19.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:20.767 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:21.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:21.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:21.769 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:11:21.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:11:21.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:21.770 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:22.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:23.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:23.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:23.724 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:11:23.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:11:23.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:23.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:24.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:25.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:25.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:25.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:11:25.606 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:11:25.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:25.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:26.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:27.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:27.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:27.056 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:11:27.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:11:27.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:27.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:11:27.475 INFO:tasks.workunit.client.0.smithi080.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1405145837,v1:127.0.0.1:6851/1405145837] [v2:127.0.0.1:6852/1405145837,v1:127.0.0.1:6853/1405145837] exists,up 8b68bba8-b275-41e9-9674-28c4f1e0a905 2024-03-25T00:11:27.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:11:27.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:11:27.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:11:27.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-03-25T00:11:27.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2024-03-25T00:11:27.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-03-25T00:11:27.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/7 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:11:27.479 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:11:27.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:11:27.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:11:27.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:11:27.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/7 2024-03-25T00:11:27.481 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:11:27.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e8c8aaf8-41ee-4681-af3b-3605a8714e62 2024-03-25T00:11:27.481 INFO:tasks.workunit.client.0.smithi080.stdout:add osd7 e8c8aaf8-41ee-4681-af3b-3605a8714e62 2024-03-25T00:11:27.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 e8c8aaf8-41ee-4681-af3b-3605a8714e62' 2024-03-25T00:11:27.482 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:11:27.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvwQBmsCRlHhAAzv1tokQ9QV/5FzbGrdpF7g== 2024-03-25T00:11:27.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvwQBmsCRlHhAAzv1tokQ9QV/5FzbGrdpF7g=="}' 2024-03-25T00:11:27.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e8c8aaf8-41ee-4681-af3b-3605a8714e62 -i td/test-erasure-code/7/new.json 2024-03-25T00:11:28.043 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:11:28.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/7/new.json 2024-03-25T00:11:28.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvwQBmsCRlHhAAzv1tokQ9QV/5FzbGrdpF7g== --osd-uuid e8c8aaf8-41ee-4681-af3b-3605a8714e62 2024-03-25T00:11:28.091 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:28.094+0000 7f77cecf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:11:28.093 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:28.098+0000 7f77cecf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:11:28.094 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:28.098+0000 7f77cecf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:11:28.095 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:28.098+0000 7f77cecf97c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-03-25T00:11:28.095 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:28.098+0000 7f77cecf97c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-03-25T00:11:28.095 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:28.098+0000 7f77cecf97c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-03-25T00:11:28.175 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:28.178+0000 7f77cecf97c0 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2024-03-25T00:11:33.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/7/keyring 2024-03-25T00:11:33.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:11:33.975 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd7 key to auth repository 2024-03-25T00:11:33.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-25T00:11:33.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:11:34.525 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.7 2024-03-25T00:11:34.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-25T00:11:34.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:11:34.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:11:34.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:11:34.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:11:34.555 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:34.558+0000 7f6dc8e5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:11:34.557 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:34.562+0000 7f6dc8e5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:11:34.558 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:34.562+0000 7f6dc8e5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:11:35.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-25T00:11:35.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:11:35.002 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:11:35.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-25T00:11:35.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:11:35.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:11:35.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:35.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:11:35.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:35.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:35.117 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:35.122+0000 7f6dc8e5e7c0 -1 Falling back to public interface 2024-03-25T00:11:35.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:36.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:36.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:36.445 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:11:36.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:11:36.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:36.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:36.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:37.689 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:11:37.694+0000 7f6dc8e5e7c0 -1 osd.7 0 log_to_monitors true 2024-03-25T00:11:37.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:37.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:37.874 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:11:37.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:11:37.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:37.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:38.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:39.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:39.644 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:11:39.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:39.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:11:39.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:39.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:40.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:41.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:41.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:41.077 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:11:41.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:11:41.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:41.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:41.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:42.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:42.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:42.482 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:11:42.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:11:42.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:42.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:43.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:44.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:44.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:44.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:11:44.121 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:11:44.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:44.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:44.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:45.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:45.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:45.537 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:11:45.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:11:45.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:45.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:46.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:47.045 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:11:47.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:11:47.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:47.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:47.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:48.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:48.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:48.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:11:48.458 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:11:48.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:48.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:48.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:49.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:49.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:49.907 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:11:49.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:11:49.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:49.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:50.618 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:51.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:51.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:51.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:11:51.621 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:11:51.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:51.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:52.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:53.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:53.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:53.230 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:11:53.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:11:53.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:53.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:54.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:55.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:55.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:55.474 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:11:55.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:11:55.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:55.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:56.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:57.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:57.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:57.276 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:11:57.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:11:57.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:57.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:58.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:11:59.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:11:59.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:11:59.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:11:59.300 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:11:59.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:11:59.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:11:59.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:00.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:00.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:00.950 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:12:00.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:12:00.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:00.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:12:01.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:02.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:02.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:02.514 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:12:02.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:12:02.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:02.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:12:02.937 INFO:tasks.workunit.client.0.smithi080.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3914457823,v1:127.0.0.1:6859/3914457823] [v2:127.0.0.1:6860/3914457823,v1:127.0.0.1:6861/3914457823] exists,up e8c8aaf8-41ee-4681-af3b-3605a8714e62 2024-03-25T00:12:02.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:12:02.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/8 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2024-03-25T00:12:02.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:12:02.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:12:02.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:12:02.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:12:02.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:12:02.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:12:02.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:12:02.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:12:02.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:12:02.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:12:02.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:12:02.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/8 2024-03-25T00:12:02.959 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:12:02.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=45313d63-6a5d-42b0-adb8-f824662d1ec9 2024-03-25T00:12:02.960 INFO:tasks.workunit.client.0.smithi080.stdout:add osd8 45313d63-6a5d-42b0-adb8-f824662d1ec9 2024-03-25T00:12:02.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 45313d63-6a5d-42b0-adb8-f824662d1ec9' 2024-03-25T00:12:02.961 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:12:02.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSwQBmKcMbOxAAJbzlRXyVPYStELJ44Svg+w== 2024-03-25T00:12:02.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSwQBmKcMbOxAAJbzlRXyVPYStELJ44Svg+w=="}' 2024-03-25T00:12:02.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 45313d63-6a5d-42b0-adb8-f824662d1ec9 -i td/test-erasure-code/8/new.json 2024-03-25T00:12:03.484 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:12:03.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/8/new.json 2024-03-25T00:12:03.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSwQBmKcMbOxAAJbzlRXyVPYStELJ44Svg+w== --osd-uuid 45313d63-6a5d-42b0-adb8-f824662d1ec9 2024-03-25T00:12:03.534 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:03.537+0000 7f3bc1e477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:03.591 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:03.593+0000 7f3bc1e477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:03.593 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:03.597+0000 7f3bc1e477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:03.593 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:03.597+0000 7f3bc1e477c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-03-25T00:12:03.593 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:03.597+0000 7f3bc1e477c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-03-25T00:12:03.593 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:03.597+0000 7f3bc1e477c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-03-25T00:12:03.651 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:03.653+0000 7f3bc1e477c0 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2024-03-25T00:12:09.978 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/8/keyring 2024-03-25T00:12:09.978 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:12:09.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-25T00:12:09.980 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd8 key to auth repository 2024-03-25T00:12:09.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:12:10.535 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.8 2024-03-25T00:12:10.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-25T00:12:10.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:12:10.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:12:10.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:12:10.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:12:10.565 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:10.569+0000 7ff86c7007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:10.619 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:10.621+0000 7ff86c7007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:10.620 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:10.625+0000 7ff86c7007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:10.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-25T00:12:10.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:12:10.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-25T00:12:10.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:12:10.939 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:12:10.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:12:10.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:10.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:12:10.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:10.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:11.170 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:11.173+0000 7ff86c7007c0 -1 Falling back to public interface 2024-03-25T00:12:11.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:12.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:12.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:12.335 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:12:12.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:12:12.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:12.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:12.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:13.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:13.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:13.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:12:13.736 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:12:13.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:13.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:13.802 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:13.805+0000 7ff86c7007c0 -1 osd.8 0 log_to_monitors true 2024-03-25T00:12:14.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:15.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:15.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:15.144 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:12:15.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:12:15.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:15.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:15.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:16.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:16.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:16.654 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:12:16.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:12:16.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:16.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:17.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:18.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:18.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:18.163 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:12:18.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:12:18.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:18.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:18.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:19.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:19.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:19.645 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:12:19.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:12:19.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:19.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:20.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:21.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:21.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:21.147 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:12:21.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:12:21.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:21.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:21.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:22.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:22.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:22.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:12:22.682 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:12:22.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:22.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:23.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:24.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:24.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:24.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:12:24.121 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:12:24.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:24.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:24.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:25.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:25.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:25.744 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:12:25.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:12:25.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:25.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:26.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:27.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:27.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:27.253 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:12:27.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:12:27.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:27.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:28.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:29.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:29.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:29.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:12:29.375 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:12:29.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:29.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:30.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:31.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:31.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:31.127 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:12:31.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:12:31.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:31.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:32.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:33.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:33.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:33.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:12:33.086 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:12:33.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:33.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:34.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:35.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:35.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:35.042 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:12:35.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:12:35.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:35.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:35.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:36.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:36.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:36.493 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:12:36.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:12:36.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:36.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:36.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:37.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:37.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:37.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:12:37.984 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:12:37.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:37.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:38.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:39.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:39.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:39.448 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:12:39.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:12:39.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:39.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:12:39.898 INFO:tasks.workunit.client.0.smithi080.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3966053311,v1:127.0.0.1:6867/3966053311] [v2:127.0.0.1:6868/3966053311,v1:127.0.0.1:6869/3966053311] exists,up 45313d63-6a5d-42b0-adb8-f824662d1ec9 2024-03-25T00:12:39.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:12:39.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:12:39.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:12:39.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-03-25T00:12:39.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2024-03-25T00:12:39.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-03-25T00:12:39.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:12:39.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-03-25T00:12:39.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:12:39.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/9 2024-03-25T00:12:39.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:12:39.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:12:39.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:12:39.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:12:39.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-03-25T00:12:39.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-03-25T00:12:39.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:12:39.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:12:39.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:12:39.905 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:12:39.905 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:12:39.905 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:12:39.905 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:12:39.906 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:12:39.906 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:12:39.906 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:12:39.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:12:39.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:12:39.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:12:39.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:12:39.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:12:39.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:12:39.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:12:39.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:12:39.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:12:39.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:12:39.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:12:39.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:12:39.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/9 2024-03-25T00:12:39.907 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:12:39.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf9c35a3-a4d9-46f1-9444-b70bca3730fe 2024-03-25T00:12:39.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 bf9c35a3-a4d9-46f1-9444-b70bca3730fe' 2024-03-25T00:12:39.907 INFO:tasks.workunit.client.0.smithi080.stdout:add osd9 bf9c35a3-a4d9-46f1-9444-b70bca3730fe 2024-03-25T00:12:39.908 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:12:39.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3wQBm8hicNxAAyvmlQaaMcNxmwePcjvkxiA== 2024-03-25T00:12:39.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3wQBm8hicNxAAyvmlQaaMcNxmwePcjvkxiA=="}' 2024-03-25T00:12:39.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf9c35a3-a4d9-46f1-9444-b70bca3730fe -i td/test-erasure-code/9/new.json 2024-03-25T00:12:40.410 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:12:40.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/9/new.json 2024-03-25T00:12:40.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3wQBm8hicNxAAyvmlQaaMcNxmwePcjvkxiA== --osd-uuid bf9c35a3-a4d9-46f1-9444-b70bca3730fe 2024-03-25T00:12:40.459 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:40.468+0000 7f7d6fa847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:40.489 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:40.496+0000 7f7d6fa847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:40.490 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:40.496+0000 7f7d6fa847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:40.491 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:40.496+0000 7f7d6fa847c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-03-25T00:12:40.491 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:40.496+0000 7f7d6fa847c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-03-25T00:12:40.491 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:40.496+0000 7f7d6fa847c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-03-25T00:12:40.528 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:40.536+0000 7f7d6fa847c0 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2024-03-25T00:12:46.833 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/9/keyring 2024-03-25T00:12:46.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:12:46.835 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd9 key to auth repository 2024-03-25T00:12:46.836 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-03-25T00:12:46.836 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:12:47.389 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.9 2024-03-25T00:12:47.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-03-25T00:12:47.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:12:47.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:12:47.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:12:47.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:12:47.415 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:47.424+0000 7f0a8619e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:47.417 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:47.424+0000 7f0a8619e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:47.418 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:47.424+0000 7f0a8619e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:12:47.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-03-25T00:12:47.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:12:47.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-25T00:12:47.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:12:47.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:12:47.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:47.828 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:12:47.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:12:47.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:47.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:12:47.977 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:47.984+0000 7f0a8619e7c0 -1 Falling back to public interface 2024-03-25T00:12:48.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:49.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:49.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:49.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:12:49.225 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:12:49.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:49.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:12:49.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:50.531 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:12:50.536+0000 7f0a8619e7c0 -1 osd.9 0 log_to_monitors true 2024-03-25T00:12:50.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:50.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:50.691 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:12:50.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:12:50.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:50.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:12:51.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:52.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:52.202 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:12:52.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:52.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:12:52.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:52.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:12:52.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:53.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:53.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:53.783 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:12:53.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:12:53.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:53.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:12:54.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:55.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:55.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:55.209 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:12:55.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:12:55.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:55.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:12:55.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:56.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:56.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:56.860 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:12:56.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:12:56.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:56.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:12:57.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:58.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:58.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:58.356 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:12:58.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:12:58.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:58.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:12:58.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:12:59.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:12:59.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:12:59.819 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:12:59.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:12:59.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:12:59.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:13:00.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:01.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:01.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:01.380 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:13:01.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:13:01.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:01.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:13:01.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:02.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:02.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:02.840 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:13:02.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:13:02.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:02.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:13:03.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:04.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:04.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:04.834 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:13:04.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:13:04.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:04.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:13:05.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:06.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:06.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:06.756 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:13:06.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:13:06.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:06.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:13:07.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:08.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:08.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:08.586 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:13:08.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:13:08.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:08.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:13:09.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:10.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:10.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:10.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:13:10.393 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:13:10.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:10.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:13:11.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:12.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:12.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:12.209 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:13:12.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:13:12.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:12.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:13:13.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:14.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:14.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:14.302 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:13:14.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:13:14.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:14.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:13:14.729 INFO:tasks.workunit.client.0.smithi080.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/3618594989,v1:127.0.0.1:6875/3618594989] [v2:127.0.0.1:6876/3618594989,v1:127.0.0.1:6877/3618594989] exists,up bf9c35a3-a4d9-46f1-9444-b70bca3730fe 2024-03-25T00:13:14.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=10 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/10 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:13:14.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:13:14.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:13:14.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:13:14.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:13:14.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:13:14.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:13:14.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:13:14.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:13:14.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:13:14.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:13:14.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:13:14.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:13:14.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/10 2024-03-25T00:13:14.736 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:13:14.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f14b7085-0367-4531-b38c-683871a1216c 2024-03-25T00:13:14.736 INFO:tasks.workunit.client.0.smithi080.stdout:add osd10 f14b7085-0367-4531-b38c-683871a1216c 2024-03-25T00:13:14.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd10 f14b7085-0367-4531-b38c-683871a1216c' 2024-03-25T00:13:14.736 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:13:14.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCawQBmUbKPLRAAGGArs5nIbC54kxdQUNrFcQ== 2024-03-25T00:13:14.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCawQBmUbKPLRAAGGArs5nIbC54kxdQUNrFcQ=="}' 2024-03-25T00:13:14.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f14b7085-0367-4531-b38c-683871a1216c -i td/test-erasure-code/10/new.json 2024-03-25T00:13:15.513 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:13:15.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/10/new.json 2024-03-25T00:13:15.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 10 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCawQBmUbKPLRAAGGArs5nIbC54kxdQUNrFcQ== --osd-uuid f14b7085-0367-4531-b38c-683871a1216c 2024-03-25T00:13:15.567 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:15.572+0000 7f6b5acc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:13:15.601 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:15.608+0000 7f6b5acc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:13:15.603 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:15.608+0000 7f6b5acc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:13:15.603 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:15.612+0000 7f6b5acc67c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-03-25T00:13:15.603 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:15.612+0000 7f6b5acc67c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-03-25T00:13:15.603 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:15.612+0000 7f6b5acc67c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-03-25T00:13:15.702 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:15.708+0000 7f6b5acc67c0 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2024-03-25T00:13:22.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/10/keyring 2024-03-25T00:13:22.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:13:22.751 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd10 key to auth repository 2024-03-25T00:13:22.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd10 key to auth repository 2024-03-25T00:13:22.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:13:23.307 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.10 2024-03-25T00:13:23.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.10 2024-03-25T00:13:23.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 10 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:13:23.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:13:23.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:13:23.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:13:23.341 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:23.348+0000 7f23e734c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:13:23.359 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:23.364+0000 7f23e734c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:13:23.360 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:23.368+0000 7f23e734c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:13:23.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 10 2024-03-25T00:13:23.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:13:23.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=10 2024-03-25T00:13:23.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:13:23.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:13:23.721 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:13:23.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:23.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:13:23.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:23.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:23.921 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:23.928+0000 7f23e734c7c0 -1 Falling back to public interface 2024-03-25T00:13:24.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:25.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:25.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:25.153 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:13:25.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:13:25.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:25.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:25.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:26.588 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:13:26.595+0000 7f23e734c7c0 -1 osd.10 0 log_to_monitors true 2024-03-25T00:13:26.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:26.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:26.612 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:13:26.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:13:26.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:26.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:27.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:28.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:28.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:28.127 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:13:28.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:13:28.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:28.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:28.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:29.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:29.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:29.758 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:13:29.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:13:29.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:29.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:30.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:31.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:31.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:31.302 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:13:31.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:13:31.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:31.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:31.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:32.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:32.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:32.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:13:32.877 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:13:32.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:32.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:33.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:34.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:34.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:34.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:13:34.509 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:13:34.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:34.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:34.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:35.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:35.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:35.935 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:13:35.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:13:35.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:35.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:36.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:37.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:37.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:37.463 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:13:37.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:13:37.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:37.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:37.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:38.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:38.983 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:13:38.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:38.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:13:38.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:38.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:39.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:40.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:40.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:40.530 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:13:40.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:13:40.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:40.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:40.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:41.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:41.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:41.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:13:41.942 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:13:41.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:41.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:43.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:44.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:44.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:44.040 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:13:44.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:13:44.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:44.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:44.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:45.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:45.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:45.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:13:45.816 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:13:45.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:45.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:47.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:48.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:48.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:48.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:13:48.034 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:13:48.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:48.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:48.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:49.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:49.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:49.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:13:49.790 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:13:49.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:49.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:50.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:51.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:51.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:51.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:13:51.865 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:13:51.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:51.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:52.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:13:53.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:13:53.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:13:53.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:13:53.794 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:13:53.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:13:53.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-03-25T00:13:54.216 INFO:tasks.workunit.client.0.smithi080.stdout:osd.10 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/2551291375,v1:127.0.0.1:6883/2551291375] [v2:127.0.0.1:6884/2551291375,v1:127.0.0.1:6885/2551291375] exists,up f14b7085-0367-4531-b38c-683871a1216c 2024-03-25T00:13:54.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:13:54.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:13:54.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:13:54.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2024-03-25T00:13:54.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-25T00:13:54.632 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' does not exist 2024-03-25T00:13:54.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-25T00:13:54.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T00:13:55.248 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' already exists 2024-03-25T00:13:55.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:13:56.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-25T00:13:59.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2024-03-25T00:13:59.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:13:59.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:13:59.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:13:59.392 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:13:59.393 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:13:59.393 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:13:59.394 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:13:59.394 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:13:59.394 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:13:59.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:13:59.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:13:59.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:13:59.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:13:59.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:13:59.515 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:14:00.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:8 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:00.071 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:14:00.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836561 2024-03-25T00:14:00.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836561 2024-03-25T00:14:00.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561' 2024-03-25T00:14:00.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:00.255 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:14:00.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673035 2024-03-25T00:14:00.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673035 2024-03-25T00:14:00.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035' 2024-03-25T00:14:00.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:00.401 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:14:00.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509507 2024-03-25T00:14:00.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509507 2024-03-25T00:14:00.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509507' 2024-03-25T00:14:00.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:00.549 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:14:00.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345986 2024-03-25T00:14:00.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345986 2024-03-25T00:14:00.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509507 3-85899345986' 2024-03-25T00:14:00.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:00.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:14:00.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182458 2024-03-25T00:14:00.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182458 2024-03-25T00:14:00.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509507 3-85899345986 4-107374182458' 2024-03-25T00:14:00.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:00.835 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:14:00.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018924 2024-03-25T00:14:00.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018924 2024-03-25T00:14:00.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509507 3-85899345986 4-107374182458 5-128849018924' 2024-03-25T00:14:00.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:00.989 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:14:01.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855396 2024-03-25T00:14:01.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855396 2024-03-25T00:14:01.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509507 3-85899345986 4-107374182458 5-128849018924 6-150323855396' 2024-03-25T00:14:01.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:01.141 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T00:14:01.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691868 2024-03-25T00:14:01.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691868 2024-03-25T00:14:01.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509507 3-85899345986 4-107374182458 5-128849018924 6-150323855396 7-171798691868' 2024-03-25T00:14:01.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:01.287 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T00:14:01.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528340 2024-03-25T00:14:01.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528340 2024-03-25T00:14:01.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509507 3-85899345986 4-107374182458 5-128849018924 6-150323855396 7-171798691868 8-193273528340' 2024-03-25T00:14:01.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:01.433 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T00:14:01.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364818 2024-03-25T00:14:01.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364818 2024-03-25T00:14:01.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509507 3-85899345986 4-107374182458 5-128849018924 6-150323855396 7-171798691868 8-193273528340 9-214748364818' 2024-03-25T00:14:01.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:01.600 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-03-25T00:14:01.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201294 2024-03-25T00:14:01.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201294 2024-03-25T00:14:01.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509507 3-85899345986 4-107374182458 5-128849018924 6-150323855396 7-171798691868 8-193273528340 9-214748364818 10-236223201294' 2024-03-25T00:14:01.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:01.751 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836561 2024-03-25T00:14:01.751 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:01.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:14:01.752 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836561 2024-03-25T00:14:01.752 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:01.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836561 2024-03-25T00:14:01.753 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 21474836561 2024-03-25T00:14:01.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-03-25T00:14:01.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:14:02.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836561 -lt 21474836561 2024-03-25T00:14:02.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:02.186 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673035 2024-03-25T00:14:02.186 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:02.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:14:02.190 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673035 2024-03-25T00:14:02.190 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:02.190 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 42949673035 2024-03-25T00:14:02.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673035 2024-03-25T00:14:02.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-03-25T00:14:02.191 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:14:02.633 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673035 -lt 42949673035 2024-03-25T00:14:02.633 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:02.634 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509507 2024-03-25T00:14:02.634 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:02.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:14:02.636 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509507 2024-03-25T00:14:02.636 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:02.638 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 64424509507 2024-03-25T00:14:02.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509507 2024-03-25T00:14:02.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-03-25T00:14:02.639 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:14:03.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509508 -lt 64424509507 2024-03-25T00:14:03.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:03.058 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345986 2024-03-25T00:14:03.058 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:03.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:14:03.060 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:03.060 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345986 2024-03-25T00:14:03.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345986 2024-03-25T00:14:03.061 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 85899345986 2024-03-25T00:14:03.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345986' 2024-03-25T00:14:03.062 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:14:03.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345986 -lt 85899345986 2024-03-25T00:14:03.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:03.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182458 2024-03-25T00:14:03.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:03.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:14:03.460 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182458 2024-03-25T00:14:03.460 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:03.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182458 2024-03-25T00:14:03.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182458' 2024-03-25T00:14:03.462 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 107374182458 2024-03-25T00:14:03.462 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:14:03.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182458 -lt 107374182458 2024-03-25T00:14:03.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:03.882 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018924 2024-03-25T00:14:03.883 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:03.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:14:03.885 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018924 2024-03-25T00:14:03.885 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:03.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018924 2024-03-25T00:14:03.886 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 128849018924 2024-03-25T00:14:03.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2024-03-25T00:14:03.886 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:14:04.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018924 -lt 128849018924 2024-03-25T00:14:04.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:04.435 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855396 2024-03-25T00:14:04.435 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:04.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:14:04.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855396 2024-03-25T00:14:04.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:04.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855396 2024-03-25T00:14:04.438 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 150323855396 2024-03-25T00:14:04.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855396' 2024-03-25T00:14:04.442 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:14:04.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855397 -lt 150323855396 2024-03-25T00:14:04.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:04.961 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691868 2024-03-25T00:14:04.962 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:04.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T00:14:04.964 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691868 2024-03-25T00:14:04.964 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:04.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691868 2024-03-25T00:14:04.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-03-25T00:14:04.965 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.7 seq 171798691868 2024-03-25T00:14:04.965 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T00:14:05.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691869 -lt 171798691868 2024-03-25T00:14:05.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:05.602 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528340 2024-03-25T00:14:05.602 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:05.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T00:14:05.605 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528340 2024-03-25T00:14:05.605 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:05.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528340 2024-03-25T00:14:05.606 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.8 seq 193273528340 2024-03-25T00:14:05.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2024-03-25T00:14:05.606 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T00:14:06.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528341 -lt 193273528340 2024-03-25T00:14:06.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:06.031 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364818 2024-03-25T00:14:06.031 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:06.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T00:14:06.033 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364818 2024-03-25T00:14:06.033 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:06.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364818 2024-03-25T00:14:06.035 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.9 seq 214748364818 2024-03-25T00:14:06.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364818' 2024-03-25T00:14:06.035 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T00:14:06.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364818 -lt 214748364818 2024-03-25T00:14:06.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:06.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201294 2024-03-25T00:14:06.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:06.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-03-25T00:14:06.561 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201294 2024-03-25T00:14:06.561 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:06.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201294 2024-03-25T00:14:06.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201294' 2024-03-25T00:14:06.563 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.10 seq 236223201294 2024-03-25T00:14:06.563 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-03-25T00:14:07.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201295 -lt 236223201294 2024-03-25T00:14:07.023 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:14:07.023 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:14:07.024 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:14:07.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-25T00:14:07.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:14:07.545 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:14:07.545 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:14:07.545 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:14:07.545 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:14:07.545 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:14:07.545 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:14:07.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T00:14:07.984 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:14:07.984 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:14:07.984 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:14:08.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-25T00:14:08.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:14:08.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:14:08.489 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2024-03-25T00:14:08.489 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:14:08.489 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:14:08.490 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:14:08.490 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:14:08.490 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:14:08.490 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48948/ceph-osd.0.asok 2024-03-25T00:14:08.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2024-03-25T00:14:08.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.48948/ceph-osd.0.asok log flush 2024-03-25T00:14:08.598 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:14:08.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2024-03-25T00:14:08.616 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:07:04.602+0000 7f0a86baa7c0 0 load: jerasure load: lrc 2024-03-25T00:14:08.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2024-03-25T00:14:08.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2024-03-25T00:14:08.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2024-03-25T00:14:09.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2024-03-25T00:14:09.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2024-03-25T00:14:10.031 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'ecpool' already exists 2024-03-25T00:14:10.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:14:11.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2024-03-25T00:14:11.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:14:11.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:14:11.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:14:11.050 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:14:11.051 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:14:11.051 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:14:11.051 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:14:11.051 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:14:11.051 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:14:11.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:14:11.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:14:11.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:14:11.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:14:11.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:14:11.171 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:8 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:14:12.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:14:12.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:12.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:14:12.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836565 2024-03-25T00:14:12.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836565 2024-03-25T00:14:12.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565' 2024-03-25T00:14:12.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:12.401 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:14:12.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673040 2024-03-25T00:14:12.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673040 2024-03-25T00:14:12.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040' 2024-03-25T00:14:12.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:12.850 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:14:13.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509512 2024-03-25T00:14:13.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509512 2024-03-25T00:14:13.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509512' 2024-03-25T00:14:13.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:13.215 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:14:13.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345990 2024-03-25T00:14:13.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345990 2024-03-25T00:14:13.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509512 3-85899345990' 2024-03-25T00:14:13.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:13.635 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:14:13.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182462 2024-03-25T00:14:13.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182462 2024-03-25T00:14:13.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509512 3-85899345990 4-107374182462' 2024-03-25T00:14:13.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:13.942 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:14:14.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018929 2024-03-25T00:14:14.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018929 2024-03-25T00:14:14.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509512 3-85899345990 4-107374182462 5-128849018929' 2024-03-25T00:14:14.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:14.317 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:14:14.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855401 2024-03-25T00:14:14.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855401 2024-03-25T00:14:14.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509512 3-85899345990 4-107374182462 5-128849018929 6-150323855401' 2024-03-25T00:14:14.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:14.473 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T00:14:14.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691873 2024-03-25T00:14:14.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691873 2024-03-25T00:14:14.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509512 3-85899345990 4-107374182462 5-128849018929 6-150323855401 7-171798691873' 2024-03-25T00:14:14.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:14.626 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T00:14:14.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528345 2024-03-25T00:14:14.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528345 2024-03-25T00:14:14.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509512 3-85899345990 4-107374182462 5-128849018929 6-150323855401 7-171798691873 8-193273528345' 2024-03-25T00:14:14.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:14.926 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T00:14:15.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364823 2024-03-25T00:14:15.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364823 2024-03-25T00:14:15.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509512 3-85899345990 4-107374182462 5-128849018929 6-150323855401 7-171798691873 8-193273528345 9-214748364823' 2024-03-25T00:14:15.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:15.282 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-03-25T00:14:15.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201299 2024-03-25T00:14:15.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201299 2024-03-25T00:14:15.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673040 2-64424509512 3-85899345990 4-107374182462 5-128849018929 6-150323855401 7-171798691873 8-193273528345 9-214748364823 10-236223201299' 2024-03-25T00:14:15.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:15.439 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836565 2024-03-25T00:14:15.439 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:15.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:14:15.441 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836565 2024-03-25T00:14:15.441 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:15.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836565 2024-03-25T00:14:15.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-03-25T00:14:15.442 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 21474836565 2024-03-25T00:14:15.443 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:14:15.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836565 2024-03-25T00:14:15.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:15.857 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673040 2024-03-25T00:14:15.857 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:15.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:14:15.859 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673040 2024-03-25T00:14:15.859 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:15.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673040 2024-03-25T00:14:15.860 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 42949673040 2024-03-25T00:14:15.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-03-25T00:14:15.861 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:14:16.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673040 -lt 42949673040 2024-03-25T00:14:16.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:16.299 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509512 2024-03-25T00:14:16.299 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:16.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:14:16.302 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:16.303 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509512 2024-03-25T00:14:16.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509512 2024-03-25T00:14:16.304 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 64424509512 2024-03-25T00:14:16.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-03-25T00:14:16.304 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:14:16.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509512 -lt 64424509512 2024-03-25T00:14:16.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:16.752 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345990 2024-03-25T00:14:16.752 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:16.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:14:16.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345990 2024-03-25T00:14:16.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:16.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345990 2024-03-25T00:14:16.756 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 85899345990 2024-03-25T00:14:16.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345990' 2024-03-25T00:14:16.756 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:14:17.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345991 -lt 85899345990 2024-03-25T00:14:17.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:17.306 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182462 2024-03-25T00:14:17.306 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:17.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:14:17.307 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182462 2024-03-25T00:14:17.308 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:17.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182462 2024-03-25T00:14:17.309 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 107374182462 2024-03-25T00:14:17.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182462' 2024-03-25T00:14:17.309 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:14:17.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182463 -lt 107374182462 2024-03-25T00:14:17.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:17.738 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018929 2024-03-25T00:14:17.739 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:17.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:14:17.740 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018929 2024-03-25T00:14:17.740 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:17.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018929 2024-03-25T00:14:17.742 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 128849018929 2024-03-25T00:14:17.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018929' 2024-03-25T00:14:17.742 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:14:18.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018929 -lt 128849018929 2024-03-25T00:14:18.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:18.138 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855401 2024-03-25T00:14:18.138 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:18.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:14:18.141 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855401 2024-03-25T00:14:18.141 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:18.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855401 2024-03-25T00:14:18.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2024-03-25T00:14:18.142 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 150323855401 2024-03-25T00:14:18.143 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:14:18.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855402 -lt 150323855401 2024-03-25T00:14:18.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:18.556 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691873 2024-03-25T00:14:18.556 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:18.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T00:14:18.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691873 2024-03-25T00:14:18.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:18.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691873 2024-03-25T00:14:18.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-03-25T00:14:18.560 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.7 seq 171798691873 2024-03-25T00:14:18.560 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T00:14:19.025 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691874 -lt 171798691873 2024-03-25T00:14:19.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:19.026 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528345 2024-03-25T00:14:19.026 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:19.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T00:14:19.029 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528345 2024-03-25T00:14:19.029 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:19.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528345 2024-03-25T00:14:19.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-03-25T00:14:19.031 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.8 seq 193273528345 2024-03-25T00:14:19.031 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T00:14:19.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528345 -lt 193273528345 2024-03-25T00:14:19.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:19.445 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364823 2024-03-25T00:14:19.445 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:19.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T00:14:19.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364823 2024-03-25T00:14:19.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:19.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364823 2024-03-25T00:14:19.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364823' 2024-03-25T00:14:19.448 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.9 seq 214748364823 2024-03-25T00:14:19.448 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T00:14:19.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364823 -lt 214748364823 2024-03-25T00:14:19.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:19.880 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201299 2024-03-25T00:14:19.881 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:19.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-03-25T00:14:19.883 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201299 2024-03-25T00:14:19.883 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:19.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201299 2024-03-25T00:14:19.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201299' 2024-03-25T00:14:19.884 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.10 seq 236223201299 2024-03-25T00:14:19.885 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-03-25T00:14:20.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201299 -lt 236223201299 2024-03-25T00:14:20.324 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:14:20.324 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:14:20.324 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:14:20.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-03-25T00:14:20.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:14:20.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:14:20.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:14:20.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:14:20.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:14:20.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:14:20.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:14:21.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-03-25T00:14:21.271 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:14:21.271 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:14:21.271 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:14:21.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-03-25T00:14:21.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:14:21.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:14:21.793 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2024-03-25T00:14:21.793 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-25T00:14:21.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2024-03-25T00:14:21.796 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_chunk_mapping 2024-03-25T00:14:21.796 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_put_get_isa 2024-03-25T00:14:21.797 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_put_get_jerasure 2024-03-25T00:14:21.797 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_put_get_lrc_advanced 2024-03-25T00:14:21.797 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_put_get_lrc_kml 2024-03-25T00:14:21.797 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_put_get_shec' 2024-03-25T00:14:21.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-03-25T00:14:21.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2024-03-25T00:14:21.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: TEST_alignment_constraints: local payload=ABC 2024-03-25T00:14:21.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: TEST_alignment_constraints: echo ABC 2024-03-25T00:14:21.798 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-03-25T00:14:21.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: local stripe_unit=4096 2024-03-25T00:14:21.816 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2024-03-25T00:14:21.816 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: grep k= 2024-03-25T00:14:22.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: eval local k=2 2024-03-25T00:14:22.244 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: local k=2 2024-03-25T00:14:22.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: local block_size=8191 2024-03-25T00:14:22.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:263: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2024-03-25T00:14:22.245 INFO:tasks.workunit.client.0.smithi080.stderr:2+0 records in 2024-03-25T00:14:22.245 INFO:tasks.workunit.client.0.smithi080.stderr:2+0 records out 2024-03-25T00:14:22.245 INFO:tasks.workunit.client.0.smithi080.stderr:16382 bytes (16 kB, 16 KiB) copied, 0.000191437 s, 85.6 MB/s 2024-03-25T00:14:22.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:264: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2024-03-25T00:14:22.295 INFO:tasks.workunit.client.0.smithi080.stderr:INFO: op_size has been rounded to 8192 2024-03-25T00:14:22.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2024-03-25T00:14:22.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-03-25T00:14:22.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2024-03-25T00:14:22.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: TEST_chunk_mapping: local dir=td/test-erasure-code 2024-03-25T00:14:22.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:311: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2024-03-25T00:14:22.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-03-25T00:14:22.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=ecpool 2024-03-25T00:14:22.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=0 2024-03-25T00:14:22.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=1 2024-03-25T00:14:22.776 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-03-25T00:14:22.776 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-03-25T00:14:22.798 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-03-25T00:14:22.799 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2024-03-25T00:14:22.799 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-03-25T00:14:22.799 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-03-25T00:14:22.820 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-03-25T00:14:22.821 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2024-03-25T00:14:22.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2024-03-25T00:14:22.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2024-03-25T00:14:22.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2024-03-25T00:14:23.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2024-03-25T00:14:23.194 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-03-25T00:14:23.194 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-25T00:14:23.194 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-03-25T00:14:23.195 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-03-25T00:14:23.195 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:14:23.618 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:14:23.619 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:14:23.619 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:14:23.619 INFO:tasks.workunit.client.0.smithi080.stderr:0' 2024-03-25T00:14:23.619 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-03-25T00:14:23.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-03-25T00:14:23.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-03-25T00:14:23.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-03-25T00:14:23.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-03-25T00:14:23.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2024-03-25T00:14:23.745 INFO:tasks.workunit.client.0.smithi080.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-03-25T00:14:23.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-03-25T00:14:23.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-03-25T00:14:23.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2024-03-25T00:14:23.870 INFO:tasks.workunit.client.0.smithi080.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-03-25T00:14:23.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-03-25T00:14:23.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-03-25T00:14:23.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-03-25T00:14:23.999 INFO:tasks.workunit.client.0.smithi080.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-03-25T00:14:24.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-03-25T00:14:24.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-03-25T00:14:24.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2024-03-25T00:14:24.123 INFO:tasks.workunit.client.0.smithi080.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-03-25T00:14:24.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-03-25T00:14:24.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-03-25T00:14:24.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:14:24.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-03-25T00:14:24.141 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-03-25T00:14:24.141 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-25T00:14:24.141 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-03-25T00:14:24.142 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-03-25T00:14:24.142 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:14:24.733 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:14:24.733 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:14:24.733 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:14:24.733 INFO:tasks.workunit.client.0.smithi080.stderr:0' 2024-03-25T00:14:24.733 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-03-25T00:14:24.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-03-25T00:14:24.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-03-25T00:14:24.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-03-25T00:14:24.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-03-25T00:14:24.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:14:24.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T00:14:24.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:14:24.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-03-25T00:14:24.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTecpool 2024-03-25T00:14:24.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-03-25T00:14:24.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:14:24.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T00:14:24.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:14:24.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2024-03-25T00:14:24.736 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:14:24.737 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:14:24.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:14:24.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:14:24.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:14:25.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:14:25.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-03-25T00:14:25.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-03-25T00:14:25.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:14:25.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T00:14:25.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:14:25.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2024-03-25T00:14:25.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2024-03-25T00:14:27.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2024-03-25T00:14:27.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-03-25T00:14:27.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:14:27.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T00:14:27.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:14:27.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/3 2024-03-25T00:14:27.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:14:27.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:14:27.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:14:27.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:14:27.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-03-25T00:14:27.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-03-25T00:14:27.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:14:27.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:14:27.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:14:27.862 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:14:27.862 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:14:27.863 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:14:27.863 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:14:27.863 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:14:27.863 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:14:27.863 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:14:27.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:14:27.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:14:27.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:14:27.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:14:27.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:14:27.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:14:27.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:14:27.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:14:27.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:14:27.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:14:27.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/3 2024-03-25T00:14:27.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T00:14:27.865 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.3 2024-03-25T00:14:27.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:14:27.865 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/3/whoami 2024-03-25T00:14:27.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T00:14:27.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:14:27.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:14:27.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:14:27.893 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:14:27.899+0000 7f4f57e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:14:27.896 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:14:27.903+0000 7f4f57e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:14:27.897 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:14:27.903+0000 7f4f57e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:14:28.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T00:14:28.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:14:28.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:14:28.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:14:28.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:14:28.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:14:28.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:14:28.277 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:14:28.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:14:28.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:14:28.448 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:14:28.455+0000 7f4f57e5f7c0 -1 Falling back to public interface 2024-03-25T00:14:28.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:14:29.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:14:29.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:14:29.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:14:29.740 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:14:29.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:14:29.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:14:30.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:14:30.963 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:14:30.966+0000 7f4f57e5f7c0 -1 osd.3 73 log_to_monitors true 2024-03-25T00:14:31.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:14:31.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:14:31.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:14:31.164 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:14:31.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:14:31.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:14:31.641 INFO:tasks.workunit.client.0.smithi080.stderr:osd.3 up in weight 1 up_from 77 up_thru 66 down_at 74 last_clean_interval [20,73) [v2:127.0.0.1:6826/503150895,v1:127.0.0.1:6827/503150895] [v2:127.0.0.1:6828/503150895,v1:127.0.0.1:6829/503150895] exists,up a585fe77-b984-4429-9348-5aa201863183 2024-03-25T00:14:31.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:14:31.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:14:31.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:14:31.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:14:31.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:14:31.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:14:31.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:14:31.643 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:14:31.643 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:14:31.643 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:14:31.643 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:14:31.643 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:14:31.643 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:14:31.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:14:31.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:14:31.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:14:31.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:14:31.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:14:31.739 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:14:32.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:14:32.264 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:14:32.264 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:14:32.264 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:14:32.265 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:14:32.265 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:14:32.265 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:14:32.265 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:14:32.265 INFO:tasks.workunit.client.0.smithi080.stderr:8 2024-03-25T00:14:32.265 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:14:32.265 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:14:32.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:14:32.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:32.266 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:14:32.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836571 2024-03-25T00:14:32.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836571 2024-03-25T00:14:32.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571' 2024-03-25T00:14:32.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:32.476 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:14:32.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673045 2024-03-25T00:14:32.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673045 2024-03-25T00:14:32.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045' 2024-03-25T00:14:32.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:32.630 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:14:32.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509518 2024-03-25T00:14:32.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509518 2024-03-25T00:14:32.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509518' 2024-03-25T00:14:32.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:32.928 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:14:33.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481795 2024-03-25T00:14:33.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481795 2024-03-25T00:14:33.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509518 3-330712481795' 2024-03-25T00:14:33.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:33.144 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:14:33.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182468 2024-03-25T00:14:33.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182468 2024-03-25T00:14:33.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509518 3-330712481795 4-107374182468' 2024-03-25T00:14:33.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:33.363 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:14:33.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018934 2024-03-25T00:14:33.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018934 2024-03-25T00:14:33.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509518 3-330712481795 4-107374182468 5-128849018934' 2024-03-25T00:14:33.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:33.516 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:14:33.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855407 2024-03-25T00:14:33.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855407 2024-03-25T00:14:33.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509518 3-330712481795 4-107374182468 5-128849018934 6-150323855407' 2024-03-25T00:14:33.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:33.671 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T00:14:33.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691879 2024-03-25T00:14:33.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691879 2024-03-25T00:14:33.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509518 3-330712481795 4-107374182468 5-128849018934 6-150323855407 7-171798691879' 2024-03-25T00:14:33.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:33.823 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T00:14:34.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528350 2024-03-25T00:14:34.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528350 2024-03-25T00:14:34.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509518 3-330712481795 4-107374182468 5-128849018934 6-150323855407 7-171798691879 8-193273528350' 2024-03-25T00:14:34.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:34.001 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T00:14:34.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364828 2024-03-25T00:14:34.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364828 2024-03-25T00:14:34.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509518 3-330712481795 4-107374182468 5-128849018934 6-150323855407 7-171798691879 8-193273528350 9-214748364828' 2024-03-25T00:14:34.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:34.151 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-03-25T00:14:34.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201305 2024-03-25T00:14:34.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201305 2024-03-25T00:14:34.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673045 2-64424509518 3-330712481795 4-107374182468 5-128849018934 6-150323855407 7-171798691879 8-193273528350 9-214748364828 10-236223201305' 2024-03-25T00:14:34.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:34.302 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836571 2024-03-25T00:14:34.302 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:34.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:14:34.303 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836571 2024-03-25T00:14:34.303 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:34.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836571 2024-03-25T00:14:34.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-03-25T00:14:34.305 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 21474836571 2024-03-25T00:14:34.305 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:14:34.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836571 2024-03-25T00:14:34.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:34.719 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673045 2024-03-25T00:14:34.719 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:34.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:14:34.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673045 2024-03-25T00:14:34.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:34.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673045 2024-03-25T00:14:34.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-03-25T00:14:34.723 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 42949673045 2024-03-25T00:14:34.724 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:14:35.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673046 -lt 42949673045 2024-03-25T00:14:35.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:35.181 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509518 2024-03-25T00:14:35.181 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:35.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:14:35.183 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509518 2024-03-25T00:14:35.183 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:35.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509518 2024-03-25T00:14:35.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-03-25T00:14:35.184 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 64424509518 2024-03-25T00:14:35.184 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:14:35.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509518 -lt 64424509518 2024-03-25T00:14:35.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:35.610 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-330712481795 2024-03-25T00:14:35.610 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:35.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:14:35.612 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-330712481795 2024-03-25T00:14:35.612 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:35.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481795 2024-03-25T00:14:35.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 330712481795' 2024-03-25T00:14:35.614 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 330712481795 2024-03-25T00:14:35.614 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:14:36.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481795 2024-03-25T00:14:36.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:36.041 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:36.041 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182468 2024-03-25T00:14:36.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:14:36.042 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:36.043 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182468 2024-03-25T00:14:36.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182468 2024-03-25T00:14:36.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-03-25T00:14:36.043 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.4 seq 107374182468 2024-03-25T00:14:36.044 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:14:36.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182469 -lt 107374182468 2024-03-25T00:14:36.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:36.444 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018934 2024-03-25T00:14:36.445 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:36.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:14:36.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018934 2024-03-25T00:14:36.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:36.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018934 2024-03-25T00:14:36.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018934' 2024-03-25T00:14:36.449 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.5 seq 128849018934 2024-03-25T00:14:36.449 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:14:36.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018935 -lt 128849018934 2024-03-25T00:14:36.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:36.863 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:36.864 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855407 2024-03-25T00:14:36.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:14:36.865 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855407 2024-03-25T00:14:36.865 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:36.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855407 2024-03-25T00:14:36.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-03-25T00:14:36.867 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.6 seq 150323855407 2024-03-25T00:14:36.867 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:14:37.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855408 -lt 150323855407 2024-03-25T00:14:37.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:37.362 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691879 2024-03-25T00:14:37.362 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:37.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T00:14:37.364 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691879 2024-03-25T00:14:37.364 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:37.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691879 2024-03-25T00:14:37.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-03-25T00:14:37.365 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.7 seq 171798691879 2024-03-25T00:14:37.365 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T00:14:37.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691879 -lt 171798691879 2024-03-25T00:14:37.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:37.805 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528350 2024-03-25T00:14:37.805 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:37.807 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T00:14:37.807 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528350 2024-03-25T00:14:37.807 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:37.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528350 2024-03-25T00:14:37.809 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-03-25T00:14:37.809 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.8 seq 193273528350 2024-03-25T00:14:37.809 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T00:14:38.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528351 -lt 193273528350 2024-03-25T00:14:38.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:38.210 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364828 2024-03-25T00:14:38.211 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:38.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T00:14:38.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364828 2024-03-25T00:14:38.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:38.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364828 2024-03-25T00:14:38.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364828' 2024-03-25T00:14:38.214 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.9 seq 214748364828 2024-03-25T00:14:38.214 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T00:14:38.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364829 -lt 214748364828 2024-03-25T00:14:38.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:38.650 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201305 2024-03-25T00:14:38.650 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:38.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-03-25T00:14:38.652 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201305 2024-03-25T00:14:38.652 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:38.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201305 2024-03-25T00:14:38.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201305' 2024-03-25T00:14:38.654 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.10 seq 236223201305 2024-03-25T00:14:38.654 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-03-25T00:14:39.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201305 -lt 236223201305 2024-03-25T00:14:39.082 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:14:39.082 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:14:39.082 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:14:39.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-03-25T00:14:39.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:14:39.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:14:39.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:14:39.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:14:39.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:14:39.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:14:39.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:14:40.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-03-25T00:14:40.141 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:14:40.142 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:14:40.142 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:14:40.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-03-25T00:14:40.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:14:40.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:14:40.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-03-25T00:14:40.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDecpool 2024-03-25T00:14:40.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-03-25T00:14:40.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:14:40.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-25T00:14:40.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:14:40.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-03-25T00:14:40.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-03-25T00:14:40.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:14:40.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-25T00:14:40.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:14:40.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2024-03-25T00:14:40.679 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:14:40.679 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:14:40.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:14:40.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:14:40.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:14:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:14:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-03-25T00:14:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-03-25T00:14:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:14:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-25T00:14:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:14:40.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2024-03-25T00:14:40.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2024-03-25T00:14:44.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2024-03-25T00:14:44.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-03-25T00:14:44.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:14:44.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-25T00:14:44.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:14:44.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/5 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:14:44.225 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:14:44.226 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:14:44.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/5 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-25T00:14:44.227 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.5 2024-03-25T00:14:44.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:14:44.228 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/5/whoami 2024-03-25T00:14:44.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-25T00:14:44.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:14:44.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:14:44.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:14:44.260 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:14:44.266+0000 7f51c2cc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:14:44.263 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:14:44.270+0000 7f51c2cc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:14:44.264 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:14:44.270+0000 7f51c2cc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:14:44.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-25T00:14:44.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:14:44.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T00:14:44.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:14:44.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:14:44.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:14:44.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:14:44.662 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:14:44.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:14:44.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:14:44.824 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:14:44.830+0000 7f51c2cc57c0 -1 Falling back to public interface 2024-03-25T00:14:45.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:14:46.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:14:46.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:14:46.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:14:46.092 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:14:46.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:14:46.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:14:46.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:14:47.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:14:47.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:14:47.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:14:47.510 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:14:47.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:14:47.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:14:47.726 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:14:47.730+0000 7f51c2cc57c0 -1 osd.5 79 log_to_monitors true 2024-03-25T00:14:48.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:14:49.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:14:49.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:14:49.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:14:49.238 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:14:49.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:14:49.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:14:50.037 INFO:tasks.workunit.client.0.smithi080.stderr:osd.5 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/910076609,v1:127.0.0.1:6843/910076609] [v2:127.0.0.1:6844/910076609,v1:127.0.0.1:6845/910076609] exists,up f3ea4677-f7e5-4cb3-982a-e71a90fb9766 2024-03-25T00:14:50.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:14:50.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:14:50.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:14:50.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:14:50.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:14:50.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:14:50.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:14:50.037 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:14:50.038 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:14:50.038 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:14:50.038 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:14:50.038 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:14:50.038 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:14:50.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:14:50.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:14:50.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:14:50.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:14:50.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:14:50.138 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:14:51.168 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:8 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:14:51.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:51.170 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:14:51.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836577 2024-03-25T00:14:51.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836577 2024-03-25T00:14:51.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577' 2024-03-25T00:14:51.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:51.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:14:51.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673051 2024-03-25T00:14:51.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673051 2024-03-25T00:14:51.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051' 2024-03-25T00:14:51.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:51.753 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:14:51.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-03-25T00:14:51.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-03-25T00:14:51.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509524' 2024-03-25T00:14:51.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:51.907 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:14:52.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481801 2024-03-25T00:14:52.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481801 2024-03-25T00:14:52.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509524 3-330712481801' 2024-03-25T00:14:52.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:52.060 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:14:52.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182474 2024-03-25T00:14:52.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182474 2024-03-25T00:14:52.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509524 3-330712481801 4-107374182474' 2024-03-25T00:14:52.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:52.215 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:14:52.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285571 2024-03-25T00:14:52.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285571 2024-03-25T00:14:52.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509524 3-330712481801 4-107374182474 5-356482285571' 2024-03-25T00:14:52.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:52.373 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:14:52.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855413 2024-03-25T00:14:52.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855413 2024-03-25T00:14:52.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509524 3-330712481801 4-107374182474 5-356482285571 6-150323855413' 2024-03-25T00:14:52.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:52.528 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T00:14:52.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691885 2024-03-25T00:14:52.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691885 2024-03-25T00:14:52.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509524 3-330712481801 4-107374182474 5-356482285571 6-150323855413 7-171798691885' 2024-03-25T00:14:52.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:52.678 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T00:14:52.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528356 2024-03-25T00:14:52.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528356 2024-03-25T00:14:52.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509524 3-330712481801 4-107374182474 5-356482285571 6-150323855413 7-171798691885 8-193273528356' 2024-03-25T00:14:52.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:52.830 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T00:14:53.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364834 2024-03-25T00:14:53.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364834 2024-03-25T00:14:53.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509524 3-330712481801 4-107374182474 5-356482285571 6-150323855413 7-171798691885 8-193273528356 9-214748364834' 2024-03-25T00:14:53.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:14:53.001 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-03-25T00:14:53.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201310 2024-03-25T00:14:53.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201310 2024-03-25T00:14:53.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509524 3-330712481801 4-107374182474 5-356482285571 6-150323855413 7-171798691885 8-193273528356 9-214748364834 10-236223201310' 2024-03-25T00:14:53.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:53.163 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:53.163 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836577 2024-03-25T00:14:53.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:14:53.164 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836577 2024-03-25T00:14:53.164 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:53.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836577 2024-03-25T00:14:53.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-03-25T00:14:53.165 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 21474836577 2024-03-25T00:14:53.165 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:14:53.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836577 -lt 21474836577 2024-03-25T00:14:53.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:53.604 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673051 2024-03-25T00:14:53.604 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:53.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:14:53.606 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673051 2024-03-25T00:14:53.609 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:53.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673051 2024-03-25T00:14:53.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-03-25T00:14:53.609 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 42949673051 2024-03-25T00:14:53.609 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:14:54.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673051 2024-03-25T00:14:54.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:54.063 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-03-25T00:14:54.063 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:54.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:14:54.066 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-03-25T00:14:54.066 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:54.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-03-25T00:14:54.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-03-25T00:14:54.068 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 64424509524 2024-03-25T00:14:54.068 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:14:54.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509524 -lt 64424509524 2024-03-25T00:14:54.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:54.479 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:54.480 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-330712481801 2024-03-25T00:14:54.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:14:54.481 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-330712481801 2024-03-25T00:14:54.481 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:54.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481801 2024-03-25T00:14:54.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 330712481801' 2024-03-25T00:14:54.482 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 330712481801 2024-03-25T00:14:54.482 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:14:54.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481801 -lt 330712481801 2024-03-25T00:14:54.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:54.919 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182474 2024-03-25T00:14:54.919 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:54.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:14:54.920 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182474 2024-03-25T00:14:54.920 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:54.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182474 2024-03-25T00:14:54.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-03-25T00:14:54.922 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.4 seq 107374182474 2024-03-25T00:14:54.922 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:14:55.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182475 -lt 107374182474 2024-03-25T00:14:55.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:55.459 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285571 2024-03-25T00:14:55.459 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:55.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:14:55.462 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285571 2024-03-25T00:14:55.462 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:55.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285571 2024-03-25T00:14:55.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285571' 2024-03-25T00:14:55.463 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.5 seq 356482285571 2024-03-25T00:14:55.463 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:14:55.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285572 -lt 356482285571 2024-03-25T00:14:55.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:55.904 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855413 2024-03-25T00:14:55.904 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:55.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:14:55.906 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855413 2024-03-25T00:14:55.906 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:55.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855413 2024-03-25T00:14:55.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2024-03-25T00:14:55.907 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.6 seq 150323855413 2024-03-25T00:14:55.907 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:14:56.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855413 -lt 150323855413 2024-03-25T00:14:56.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:56.362 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691885 2024-03-25T00:14:56.362 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:56.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T00:14:56.364 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691885 2024-03-25T00:14:56.365 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:56.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691885 2024-03-25T00:14:56.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691885' 2024-03-25T00:14:56.367 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.7 seq 171798691885 2024-03-25T00:14:56.367 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T00:14:56.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691885 -lt 171798691885 2024-03-25T00:14:56.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:56.794 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528356 2024-03-25T00:14:56.794 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:56.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T00:14:56.795 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528356 2024-03-25T00:14:56.795 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:56.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528356 2024-03-25T00:14:56.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2024-03-25T00:14:56.797 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.8 seq 193273528356 2024-03-25T00:14:56.797 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T00:14:57.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528357 -lt 193273528356 2024-03-25T00:14:57.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:57.284 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364834 2024-03-25T00:14:57.285 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:57.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T00:14:57.288 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364834 2024-03-25T00:14:57.288 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:57.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364834 2024-03-25T00:14:57.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364834' 2024-03-25T00:14:57.289 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.9 seq 214748364834 2024-03-25T00:14:57.289 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T00:14:57.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364835 -lt 214748364834 2024-03-25T00:14:57.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:14:57.747 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201310 2024-03-25T00:14:57.747 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:14:57.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-03-25T00:14:57.749 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201310 2024-03-25T00:14:57.749 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:14:57.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201310 2024-03-25T00:14:57.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201310' 2024-03-25T00:14:57.750 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.10 seq 236223201310 2024-03-25T00:14:57.750 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-03-25T00:14:58.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201311 -lt 236223201310 2024-03-25T00:14:58.195 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:14:58.195 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:14:58.196 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:14:58.747 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-03-25T00:14:58.747 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:14:58.748 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:14:58.748 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:14:58.748 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:14:58.748 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:14:58.748 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:14:58.749 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:14:59.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-03-25T00:14:59.138 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:14:59.139 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:14:59.139 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:14:59.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-03-25T00:14:59.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:14:59.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:14:59.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:313: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2024-03-25T00:15:00.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:318: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2024-03-25T00:15:00.769 INFO:tasks.workunit.client.0.smithi080.stdout:crush-device-class= 2024-03-25T00:15:00.770 INFO:tasks.workunit.client.0.smithi080.stdout:crush-failure-domain=host 2024-03-25T00:15:00.770 INFO:tasks.workunit.client.0.smithi080.stdout:crush-num-failure-domains=0 2024-03-25T00:15:00.770 INFO:tasks.workunit.client.0.smithi080.stdout:crush-osds-per-failure-domain=0 2024-03-25T00:15:00.770 INFO:tasks.workunit.client.0.smithi080.stdout:crush-root=default 2024-03-25T00:15:00.770 INFO:tasks.workunit.client.0.smithi080.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2024-03-25T00:15:00.770 INFO:tasks.workunit.client.0.smithi080.stdout:k=-1 2024-03-25T00:15:00.770 INFO:tasks.workunit.client.0.smithi080.stdout:l=-1 2024-03-25T00:15:00.770 INFO:tasks.workunit.client.0.smithi080.stdout:layers=[ [ "cDD", "" ] ] 2024-03-25T00:15:00.770 INFO:tasks.workunit.client.0.smithi080.stdout:m=-1 2024-03-25T00:15:00.770 INFO:tasks.workunit.client.0.smithi080.stdout:mapping=_DD 2024-03-25T00:15:00.770 INFO:tasks.workunit.client.0.smithi080.stdout:plugin=lrc 2024-03-25T00:15:00.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:319: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2024-03-25T00:15:00.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2024-03-25T00:15:01.754 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'remap-pool' already exists 2024-03-25T00:15:01.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:15:02.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:327: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2024-03-25T00:15:02.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-03-25T00:15:02.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=remap-pool 2024-03-25T00:15:02.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=1 2024-03-25T00:15:02.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=2 2024-03-25T00:15:02.776 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-03-25T00:15:02.777 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-03-25T00:15:02.799 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-03-25T00:15:02.799 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2024-03-25T00:15:02.799 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-03-25T00:15:02.800 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-03-25T00:15:02.822 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-03-25T00:15:02.823 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2024-03-25T00:15:02.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2024-03-25T00:15:02.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2024-03-25T00:15:02.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2024-03-25T00:15:04.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2024-03-25T00:15:05.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-03-25T00:15:05.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-03-25T00:15:05.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-03-25T00:15:05.250 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-03-25T00:15:05.250 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:15:05.889 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-03-25T00:15:05.890 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:15:05.890 INFO:tasks.workunit.client.0.smithi080.stderr:9' 2024-03-25T00:15:05.890 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-03-25T00:15:05.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('4' '6' '9') 2024-03-25T00:15:05.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-03-25T00:15:05.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-03-25T00:15:05.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-03-25T00:15:05.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2024-03-25T00:15:06.016 INFO:tasks.workunit.client.0.smithi080.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-03-25T00:15:06.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-03-25T00:15:06.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-03-25T00:15:06.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2024-03-25T00:15:06.145 INFO:tasks.workunit.client.0.smithi080.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-03-25T00:15:06.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-03-25T00:15:06.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-03-25T00:15:06.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-03-25T00:15:06.274 INFO:tasks.workunit.client.0.smithi080.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-03-25T00:15:06.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-03-25T00:15:06.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-03-25T00:15:06.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:15:06.278 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-03-25T00:15:06.279 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-03-25T00:15:06.279 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-03-25T00:15:06.279 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-03-25T00:15:06.280 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-03-25T00:15:06.280 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:15:06.709 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-03-25T00:15:06.709 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:15:06.709 INFO:tasks.workunit.client.0.smithi080.stderr:9' 2024-03-25T00:15:06.709 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('4' '6' '9') 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=6 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=6 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2024-03-25T00:15:06.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2024-03-25T00:15:06.711 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:15:06.711 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:15:06.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:15:06.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:15:06.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:15:08.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:15:08.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-03-25T00:15:08.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-03-25T00:15:08.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:15:08.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=6 2024-03-25T00:15:08.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:15:08.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2024-03-25T00:15:08.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2024-03-25T00:15:11.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2024-03-25T00:15:11.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-03-25T00:15:11.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:15:11.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=6 2024-03-25T00:15:11.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:15:11.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/6 2024-03-25T00:15:11.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:15:11.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:15:11.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:15:11.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:15:11.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-03-25T00:15:11.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-03-25T00:15:11.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:15:11.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:15:11.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:15:11.027 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:15:11.027 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:15:11.027 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:15:11.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:15:11.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:15:11.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/6 2024-03-25T00:15:11.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.6 2024-03-25T00:15:11.031 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.6 2024-03-25T00:15:11.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 6 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:15:11.031 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/6/whoami 2024-03-25T00:15:11.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 6 = 6 ']' 2024-03-25T00:15:11.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:15:11.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:15:11.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:15:11.061 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:15:11.066+0000 7fb742f167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:15:11.064 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:15:11.070+0000 7fb742f167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:15:11.066 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:15:11.070+0000 7fb742f167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:15:11.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 6 2024-03-25T00:15:11.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:15:11.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-25T00:15:11.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:15:11.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:15:11.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:15:11.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:15:11.449 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:15:11.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:15:11.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:15:11.644 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:15:11.650+0000 7fb742f167c0 -1 Falling back to public interface 2024-03-25T00:15:11.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:15:12.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:15:12.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:15:12.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:15:12.929 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:15:12.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:15:12.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:15:13.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:15:14.201 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:15:14.206+0000 7fb742f167c0 -1 osd.6 95 log_to_monitors true 2024-03-25T00:15:14.350 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:15:14.350 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:15:14.350 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:15:14.350 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:15:14.350 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:15:14.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:15:15.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:15:16.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:15:16.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:15:16.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:15:16.029 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:15:16.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:15:16.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:15:16.844 INFO:tasks.workunit.client.0.smithi080.stderr:osd.6 up in weight 1 up_from 101 up_thru 101 down_at 97 last_clean_interval [35,95) [v2:127.0.0.1:6850/3291775194,v1:127.0.0.1:6851/3291775194] [v2:127.0.0.1:6852/3291775194,v1:127.0.0.1:6853/3291775194] exists,up 8b68bba8-b275-41e9-9674-28c4f1e0a905 2024-03-25T00:15:16.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:15:16.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:15:16.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:15:16.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:15:16.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:15:16.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:15:16.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:15:16.846 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:15:16.846 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:15:16.846 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:15:16.846 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:15:16.846 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:15:16.846 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:15:16.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:15:16.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:15:16.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:15:16.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:15:16.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:15:16.963 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:15:17.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:15:17.652 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:15:17.652 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:15:17.652 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:15:17.652 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:15:17.652 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:15:17.653 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:15:17.653 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:15:17.653 INFO:tasks.workunit.client.0.smithi080.stderr:8 2024-03-25T00:15:17.653 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:15:17.653 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:15:17.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:15:17.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:17.653 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:15:17.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836584 2024-03-25T00:15:17.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836584 2024-03-25T00:15:17.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584' 2024-03-25T00:15:17.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:17.855 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:15:18.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-03-25T00:15:18.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-03-25T00:15:18.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059' 2024-03-25T00:15:18.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:18.105 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:15:18.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509531 2024-03-25T00:15:18.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509531 2024-03-25T00:15:18.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509531' 2024-03-25T00:15:18.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:18.259 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:15:18.412 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481808 2024-03-25T00:15:18.412 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481808 2024-03-25T00:15:18.413 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509531 3-330712481808' 2024-03-25T00:15:18.413 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:18.413 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:15:18.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182481 2024-03-25T00:15:18.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182481 2024-03-25T00:15:18.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509531 3-330712481808 4-107374182481' 2024-03-25T00:15:18.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:18.564 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:15:18.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285579 2024-03-25T00:15:18.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285579 2024-03-25T00:15:18.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509531 3-330712481808 4-107374182481 5-356482285579' 2024-03-25T00:15:18.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:18.713 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:15:18.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696899 2024-03-25T00:15:18.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696899 2024-03-25T00:15:18.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509531 3-330712481808 4-107374182481 5-356482285579 6-433791696899' 2024-03-25T00:15:18.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:18.864 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T00:15:19.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691892 2024-03-25T00:15:19.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691892 2024-03-25T00:15:19.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509531 3-330712481808 4-107374182481 5-356482285579 6-433791696899 7-171798691892' 2024-03-25T00:15:19.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:19.014 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T00:15:19.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528363 2024-03-25T00:15:19.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528363 2024-03-25T00:15:19.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509531 3-330712481808 4-107374182481 5-356482285579 6-433791696899 7-171798691892 8-193273528363' 2024-03-25T00:15:19.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:19.175 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T00:15:19.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364841 2024-03-25T00:15:19.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364841 2024-03-25T00:15:19.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509531 3-330712481808 4-107374182481 5-356482285579 6-433791696899 7-171798691892 8-193273528363 9-214748364841' 2024-03-25T00:15:19.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:19.328 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-03-25T00:15:19.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201317 2024-03-25T00:15:19.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201317 2024-03-25T00:15:19.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509531 3-330712481808 4-107374182481 5-356482285579 6-433791696899 7-171798691892 8-193273528363 9-214748364841 10-236223201317' 2024-03-25T00:15:19.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:19.479 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836584 2024-03-25T00:15:19.479 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:19.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:15:19.480 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836584 2024-03-25T00:15:19.480 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:19.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836584 2024-03-25T00:15:19.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-03-25T00:15:19.481 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 21474836584 2024-03-25T00:15:19.482 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:15:19.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836584 2024-03-25T00:15:19.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:19.885 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-03-25T00:15:19.885 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:19.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:15:19.887 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-03-25T00:15:19.888 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:19.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-03-25T00:15:19.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-03-25T00:15:19.890 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 42949673059 2024-03-25T00:15:19.890 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:15:20.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673059 -lt 42949673059 2024-03-25T00:15:20.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:20.317 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509531 2024-03-25T00:15:20.318 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:20.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:15:20.319 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509531 2024-03-25T00:15:20.319 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:20.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509531 2024-03-25T00:15:20.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-03-25T00:15:20.321 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 64424509531 2024-03-25T00:15:20.321 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:15:20.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509531 -lt 64424509531 2024-03-25T00:15:20.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:20.716 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-330712481808 2024-03-25T00:15:20.716 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:20.717 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:15:20.718 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:20.718 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-330712481808 2024-03-25T00:15:20.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481808 2024-03-25T00:15:20.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 330712481808' 2024-03-25T00:15:20.719 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 330712481808 2024-03-25T00:15:20.719 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:15:21.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481808 -lt 330712481808 2024-03-25T00:15:21.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:21.239 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182481 2024-03-25T00:15:21.240 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:21.241 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:15:21.242 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182481 2024-03-25T00:15:21.242 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:21.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182481 2024-03-25T00:15:21.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-03-25T00:15:21.243 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.4 seq 107374182481 2024-03-25T00:15:21.244 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:15:21.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182482 -lt 107374182481 2024-03-25T00:15:21.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:21.694 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285579 2024-03-25T00:15:21.694 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:21.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:15:21.696 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285579 2024-03-25T00:15:21.696 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:21.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285579 2024-03-25T00:15:21.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285579' 2024-03-25T00:15:21.697 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.5 seq 356482285579 2024-03-25T00:15:21.697 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:15:22.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285579 -lt 356482285579 2024-03-25T00:15:22.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:22.168 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-433791696899 2024-03-25T00:15:22.168 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:22.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:15:22.170 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-433791696899 2024-03-25T00:15:22.170 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:22.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696899 2024-03-25T00:15:22.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 433791696899' 2024-03-25T00:15:22.171 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.6 seq 433791696899 2024-03-25T00:15:22.171 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:15:22.603 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696899 2024-03-25T00:15:22.603 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:22.604 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691892 2024-03-25T00:15:22.604 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:22.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T00:15:22.605 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691892 2024-03-25T00:15:22.606 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:22.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691892 2024-03-25T00:15:22.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-03-25T00:15:22.607 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.7 seq 171798691892 2024-03-25T00:15:22.607 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T00:15:23.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691893 -lt 171798691892 2024-03-25T00:15:23.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:23.054 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528363 2024-03-25T00:15:23.054 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:23.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T00:15:23.056 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528363 2024-03-25T00:15:23.056 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:23.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528363 2024-03-25T00:15:23.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528363' 2024-03-25T00:15:23.058 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.8 seq 193273528363 2024-03-25T00:15:23.058 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T00:15:23.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528364 -lt 193273528363 2024-03-25T00:15:23.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:23.475 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364841 2024-03-25T00:15:23.475 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:23.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T00:15:23.478 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364841 2024-03-25T00:15:23.478 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:23.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364841 2024-03-25T00:15:23.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364841' 2024-03-25T00:15:23.479 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.9 seq 214748364841 2024-03-25T00:15:23.479 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T00:15:23.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364842 -lt 214748364841 2024-03-25T00:15:23.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:23.897 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201317 2024-03-25T00:15:23.897 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:23.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-03-25T00:15:23.900 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:23.900 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201317 2024-03-25T00:15:23.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201317 2024-03-25T00:15:23.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201317' 2024-03-25T00:15:23.902 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.10 seq 236223201317 2024-03-25T00:15:23.902 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-03-25T00:15:24.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201318 -lt 236223201317 2024-03-25T00:15:24.328 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:15:24.328 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:15:24.328 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:15:24.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-03-25T00:15:24.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:15:24.892 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:15:24.892 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:15:24.892 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:15:24.892 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:15:24.893 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:15:24.893 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:15:25.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-03-25T00:15:25.289 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:15:25.289 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:15:25.289 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:15:25.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-03-25T00:15:25.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:15:25.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:15:25.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-03-25T00:15:25.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-03-25T00:15:25.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:15:25.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-25T00:15:25.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:15:25.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-03-25T00:15:25.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-03-25T00:15:25.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:15:25.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-25T00:15:25.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:15:25.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2024-03-25T00:15:25.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDremap-pool 2024-03-25T00:15:25.814 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:15:25.814 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:15:25.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:15:25.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:15:25.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:15:27.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:15:27.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-03-25T00:15:27.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-03-25T00:15:27.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:15:27.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-25T00:15:27.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:15:27.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2024-03-25T00:15:27.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2024-03-25T00:15:29.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2024-03-25T00:15:29.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-03-25T00:15:29.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:15:29.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-25T00:15:29.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:15:29.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/9 2024-03-25T00:15:29.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-03-25T00:15:29.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:15:29.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:15:29.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:15:29.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-03-25T00:15:29.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-03-25T00:15:29.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:15:29.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:15:29.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-03-25T00:15:29.804 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:15:29.804 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:15:29.804 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:15:29.804 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:15:29.805 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:15:29.805 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:15:29.805 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:15:29.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' 2024-03-25T00:15:29.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:15:29.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-03-25T00:15:29.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-03-25T00:15:29.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:15:29.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:15:29.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:15:29.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:15:29.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:15:29.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:15:29.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/9 2024-03-25T00:15:29.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-25T00:15:29.806 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.9 2024-03-25T00:15:29.807 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=297ce0d0-9d27-4f53-8582-5eae2bccbc36 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48948/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:15:29.807 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/9/whoami 2024-03-25T00:15:29.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-25T00:15:29.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:15:29.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:15:29.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:15:29.835 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:15:29.838+0000 7fe70f0007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:15:29.837 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:15:29.842+0000 7fe70f0007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:15:29.839 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:15:29.846+0000 7fe70f0007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:15:30.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-25T00:15:30.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:15:30.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-25T00:15:30.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:15:30.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:15:30.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:15:30.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:15:30.230 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:15:30.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:15:30.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:15:30.408 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:15:30.414+0000 7fe70f0007c0 -1 Falling back to public interface 2024-03-25T00:15:30.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:15:31.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:15:31.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:15:31.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:15:31.644 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:15:31.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:15:31.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:15:32.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:15:32.750 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:15:32.754+0000 7fe70f0007c0 -1 osd.9 103 log_to_monitors true 2024-03-25T00:15:33.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:15:33.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:15:33.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:15:33.051 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:15:33.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:15:33.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:15:33.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:15:34.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:15:34.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:15:34.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:15:34.462 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:15:34.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:15:34.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-25T00:15:35.089 INFO:tasks.workunit.client.0.smithi080.stderr:osd.9 up in weight 1 up_from 108 up_thru 108 down_at 104 last_clean_interval [50,103) [v2:127.0.0.1:6874/2919183475,v1:127.0.0.1:6875/2919183475] [v2:127.0.0.1:6876/2919183475,v1:127.0.0.1:6877/2919183475] exists,up bf9c35a3-a4d9-46f1-9444-b70bca3730fe 2024-03-25T00:15:35.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:15:35.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:15:35.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:15:35.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:15:35.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:15:35.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:15:35.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:15:35.091 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:15:35.091 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:15:35.091 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:15:35.091 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:15:35.091 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:15:35.091 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:15:35.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:15:35.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:15:35.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:15:35.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:15:35.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:15:35.191 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:15:35.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:15:35.936 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:15:35.936 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:15:35.936 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:15:35.936 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:15:35.936 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:15:35.936 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:15:35.936 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:15:35.936 INFO:tasks.workunit.client.0.smithi080.stderr:8 2024-03-25T00:15:35.937 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:15:35.937 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:15:35.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:15:35.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:35.937 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:15:36.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836590 2024-03-25T00:15:36.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836590 2024-03-25T00:15:36.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590' 2024-03-25T00:15:36.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:36.256 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:15:36.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673064 2024-03-25T00:15:36.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673064 2024-03-25T00:15:36.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673064' 2024-03-25T00:15:36.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:36.410 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:15:36.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509537 2024-03-25T00:15:36.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509537 2024-03-25T00:15:36.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673064 2-64424509537' 2024-03-25T00:15:36.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:36.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:15:36.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481814 2024-03-25T00:15:36.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481814 2024-03-25T00:15:36.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673064 2-64424509537 3-330712481814' 2024-03-25T00:15:36.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:36.830 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:15:36.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182487 2024-03-25T00:15:36.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182487 2024-03-25T00:15:36.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673064 2-64424509537 3-330712481814 4-107374182487' 2024-03-25T00:15:36.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:36.979 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:15:37.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285584 2024-03-25T00:15:37.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285584 2024-03-25T00:15:37.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673064 2-64424509537 3-330712481814 4-107374182487 5-356482285584' 2024-03-25T00:15:37.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:37.129 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:15:37.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696905 2024-03-25T00:15:37.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696905 2024-03-25T00:15:37.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673064 2-64424509537 3-330712481814 4-107374182487 5-356482285584 6-433791696905' 2024-03-25T00:15:37.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:37.281 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T00:15:37.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691897 2024-03-25T00:15:37.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691897 2024-03-25T00:15:37.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673064 2-64424509537 3-330712481814 4-107374182487 5-356482285584 6-433791696905 7-171798691897' 2024-03-25T00:15:37.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:37.430 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T00:15:37.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-03-25T00:15:37.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-03-25T00:15:37.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673064 2-64424509537 3-330712481814 4-107374182487 5-356482285584 6-433791696905 7-171798691897 8-193273528369' 2024-03-25T00:15:37.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:37.577 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T00:15:37.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467971 2024-03-25T00:15:37.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467971 2024-03-25T00:15:37.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673064 2-64424509537 3-330712481814 4-107374182487 5-356482285584 6-433791696905 7-171798691897 8-193273528369 9-463856467971' 2024-03-25T00:15:37.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:15:37.730 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-03-25T00:15:37.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201323 2024-03-25T00:15:37.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201323 2024-03-25T00:15:37.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673064 2-64424509537 3-330712481814 4-107374182487 5-356482285584 6-433791696905 7-171798691897 8-193273528369 9-463856467971 10-236223201323' 2024-03-25T00:15:37.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:37.881 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836590 2024-03-25T00:15:37.881 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:37.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:15:37.882 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836590 2024-03-25T00:15:37.882 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:37.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836590 2024-03-25T00:15:37.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2024-03-25T00:15:37.883 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 21474836590 2024-03-25T00:15:37.884 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:15:38.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836590 -lt 21474836590 2024-03-25T00:15:38.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:38.281 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673064 2024-03-25T00:15:38.281 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:38.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:15:38.283 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673064 2024-03-25T00:15:38.283 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:38.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673064 2024-03-25T00:15:38.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2024-03-25T00:15:38.284 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 42949673064 2024-03-25T00:15:38.285 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:15:38.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673064 -lt 42949673064 2024-03-25T00:15:38.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:38.734 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509537 2024-03-25T00:15:38.734 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:38.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:15:38.736 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509537 2024-03-25T00:15:38.736 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:38.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509537 2024-03-25T00:15:38.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-03-25T00:15:38.737 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 64424509537 2024-03-25T00:15:38.737 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:15:39.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509537 2024-03-25T00:15:39.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:39.154 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:39.154 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-330712481814 2024-03-25T00:15:39.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:15:39.156 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-330712481814 2024-03-25T00:15:39.156 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:39.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481814 2024-03-25T00:15:39.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 330712481814' 2024-03-25T00:15:39.157 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 330712481814 2024-03-25T00:15:39.157 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:15:39.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481814 -lt 330712481814 2024-03-25T00:15:39.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:39.593 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182487 2024-03-25T00:15:39.593 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:39.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:15:39.595 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182487 2024-03-25T00:15:39.596 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:39.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182487 2024-03-25T00:15:39.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2024-03-25T00:15:39.597 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.4 seq 107374182487 2024-03-25T00:15:39.597 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:15:40.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182487 -lt 107374182487 2024-03-25T00:15:40.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:40.018 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285584 2024-03-25T00:15:40.018 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:40.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:15:40.020 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285584 2024-03-25T00:15:40.020 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285584 2024-03-25T00:15:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285584' 2024-03-25T00:15:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.5 seq 356482285584 2024-03-25T00:15:40.021 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:15:40.437 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285585 -lt 356482285584 2024-03-25T00:15:40.437 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:40.438 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-433791696905 2024-03-25T00:15:40.438 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:40.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:15:40.440 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-433791696905 2024-03-25T00:15:40.440 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:40.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696905 2024-03-25T00:15:40.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 433791696905' 2024-03-25T00:15:40.441 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.6 seq 433791696905 2024-03-25T00:15:40.441 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:15:40.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696905 -lt 433791696905 2024-03-25T00:15:40.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:40.888 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:40.888 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691897 2024-03-25T00:15:40.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T00:15:40.890 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691897 2024-03-25T00:15:40.890 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:40.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691897 2024-03-25T00:15:40.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-03-25T00:15:40.892 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.7 seq 171798691897 2024-03-25T00:15:40.892 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T00:15:41.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691898 -lt 171798691897 2024-03-25T00:15:41.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:41.336 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-03-25T00:15:41.336 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:41.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T00:15:41.338 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-03-25T00:15:41.338 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:41.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-03-25T00:15:41.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-03-25T00:15:41.339 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.8 seq 193273528369 2024-03-25T00:15:41.340 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T00:15:41.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528369 2024-03-25T00:15:41.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:41.736 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-463856467971 2024-03-25T00:15:41.736 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:41.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T00:15:41.739 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-463856467971 2024-03-25T00:15:41.739 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:41.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467971 2024-03-25T00:15:41.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 463856467971' 2024-03-25T00:15:41.739 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.9 seq 463856467971 2024-03-25T00:15:41.740 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T00:15:42.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467972 -lt 463856467971 2024-03-25T00:15:42.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:15:42.172 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201323 2024-03-25T00:15:42.172 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:15:42.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-03-25T00:15:42.175 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201323 2024-03-25T00:15:42.175 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:15:42.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201323 2024-03-25T00:15:42.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201323' 2024-03-25T00:15:42.177 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.10 seq 236223201323 2024-03-25T00:15:42.177 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-03-25T00:15:42.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201324 -lt 236223201323 2024-03-25T00:15:42.625 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:15:42.625 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:15:42.625 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:15:43.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-03-25T00:15:43.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:15:43.158 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:15:43.158 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:15:43.158 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:15:43.158 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:15:43.158 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:15:43.158 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:15:43.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-03-25T00:15:43.582 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:15:43.583 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:15:43.583 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:15:44.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-03-25T00:15:44.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:15:44.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:15:44.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:329: TEST_chunk_mapping: delete_pool remap-pool 2024-03-25T00:15:44.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=remap-pool 2024-03-25T00:15:44.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2024-03-25T00:15:44.612 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'remap-pool' does not exist 2024-03-25T00:15:44.627 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:330: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2024-03-25T00:15:45.602 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile remap-profile does not exist 2024-03-25T00:15:45.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-03-25T00:15:45.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2024-03-25T00:15:45.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2024-03-25T00:15:45.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: erasure_code_plugin_exists: local plugin=isa 2024-03-25T00:15:45.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: erasure_code_plugin_exists: local status 2024-03-25T00:15:45.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: erasure_code_plugin_exists: local grepstr 2024-03-25T00:15:45.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: erasure_code_plugin_exists: local s 2024-03-25T00:15:45.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: case `uname` in 2024-03-25T00:15:45.621 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: uname 2024-03-25T00:15:45.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: erasure_code_plugin_exists: grepstr='isa.*No such file' 2024-03-25T00:15:45.623 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2024-03-25T00:15:46.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: s= 2024-03-25T00:15:46.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local status=0 2024-03-25T00:15:46.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-03-25T00:15:46.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-03-25T00:15:47.429 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile TESTPROFILE does not exist 2024-03-25T00:15:47.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: return 0 2024-03-25T00:15:47.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:197: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2024-03-25T00:15:47.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:198: TEST_rados_put_get_isa: local poolname=pool-isa 2024-03-25T00:15:47.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:200: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2024-03-25T00:15:48.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:203: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2024-03-25T00:15:48.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2024-03-25T00:15:49.423 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-isa' already exists 2024-03-25T00:15:49.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:15:50.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:206: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2024-03-25T00:15:50.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-03-25T00:15:50.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2024-03-25T00:15:50.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-03-25T00:15:50.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:15:50.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-03-25T00:15:50.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:15:50.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-03-25T00:15:50.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:15:50.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-03-25T00:15:50.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:15:50.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-03-25T00:15:50.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2024-03-25T00:15:52.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:15:52.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:15:52.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-03-25T00:15:52.391 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2024-03-25T00:15:52.391 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-03-25T00:15:52.391 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:15:52.392 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-03-25T00:15:52.392 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:15:52.858 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-03-25T00:15:52.859 INFO:tasks.workunit.client.0.smithi080.stderr:10 2024-03-25T00:15:52.859 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:15:52.859 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:15:52.859 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:15:52.859 INFO:tasks.workunit.client.0.smithi080.stderr:8 2024-03-25T00:15:52.859 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:15:52.859 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:15:52.859 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:15:52.859 INFO:tasks.workunit.client.0.smithi080.stderr:9' 2024-03-25T00:15:52.859 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2024-03-25T00:15:52.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('4' '10' '0' '7' '6' '8' '2' '1' '5' '9') 2024-03-25T00:15:52.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-03-25T00:15:52.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2024-03-25T00:15:52.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-03-25T00:15:53.640 INFO:tasks.workunit.client.0.smithi080.stderr:osd.9 is already out. 2024-03-25T00:15:53.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-03-25T00:15:58.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-isa SOMETHING 2024-03-25T00:15:58.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-03-25T00:15:58.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:15:58.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-03-25T00:15:58.659 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-03-25T00:15:58.660 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:15:59.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-03-25T00:15:59.090 INFO:tasks.workunit.client.0.smithi080.stderr:10 2024-03-25T00:15:59.090 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:15:59.090 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:15:59.090 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:15:59.090 INFO:tasks.workunit.client.0.smithi080.stderr:8 2024-03-25T00:15:59.090 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:15:59.090 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:15:59.091 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:15:59.091 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-25T00:15:59.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2024-03-25T00:15:59.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:15:59.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:15:59.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-03-25T00:15:59.850 INFO:tasks.workunit.client.0.smithi080.stderr:osd.9 is already in. 2024-03-25T00:15:59.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-03-25T00:15:59.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_isa: delete_pool pool-isa 2024-03-25T00:15:59.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-isa 2024-03-25T00:15:59.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2024-03-25T00:16:01.490 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-isa' does not exist 2024-03-25T00:16:01.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-03-25T00:16:01.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2024-03-25T00:16:01.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2024-03-25T00:16:01.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:214: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2024-03-25T00:16:01.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-03-25T00:16:01.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2024-03-25T00:16:01.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-03-25T00:16:01.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:16:01.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-03-25T00:16:01.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:16:01.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-03-25T00:16:01.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:16:01.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-03-25T00:16:01.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:16:01.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-03-25T00:16:01.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2024-03-25T00:16:02.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:16:03.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:16:03.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-03-25T00:16:03.226 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2024-03-25T00:16:03.226 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-25T00:16:03.226 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:16:03.226 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-25T00:16:03.227 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:16:03.829 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-03-25T00:16:03.829 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:16:03.829 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:16:03.829 INFO:tasks.workunit.client.0.smithi080.stderr:9' 2024-03-25T00:16:03.829 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 9 2024-03-25T00:16:03.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('6' '0' '7' '9') 2024-03-25T00:16:03.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-03-25T00:16:03.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2024-03-25T00:16:03.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-03-25T00:16:04.358 INFO:tasks.workunit.client.0.smithi080.stderr:osd.9 is already out. 2024-03-25T00:16:04.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-03-25T00:16:09.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds ecpool SOMETHING 2024-03-25T00:16:09.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-03-25T00:16:09.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-25T00:16:09.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:16:09.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-25T00:16:09.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:16:09.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-03-25T00:16:09.855 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:16:09.855 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:16:09.855 INFO:tasks.workunit.client.0.smithi080.stderr:5' 2024-03-25T00:16:09.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 5 2024-03-25T00:16:09.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:16:09.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:16:09.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-03-25T00:16:10.488 INFO:tasks.workunit.client.0.smithi080.stderr:osd.9 is already in. 2024-03-25T00:16:10.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-03-25T00:16:10.504 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:216: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2024-03-25T00:16:10.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:217: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2024-03-25T00:16:10.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2024-03-25T00:16:11.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2024-03-25T00:16:11.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2024-03-25T00:16:13.486 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' already exists 2024-03-25T00:16:13.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:16:14.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2024-03-25T00:16:14.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-03-25T00:16:14.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2024-03-25T00:16:14.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-03-25T00:16:14.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:16:14.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-03-25T00:16:14.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:16:14.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-03-25T00:16:14.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:16:14.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-03-25T00:16:14.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:16:14.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-03-25T00:16:14.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-03-25T00:16:16.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:16:17.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:16:17.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-03-25T00:16:17.058 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2024-03-25T00:16:17.058 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:16:17.058 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:16:17.059 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-03-25T00:16:17.059 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:16:17.646 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-03-25T00:16:17.646 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:16:17.646 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:16:17.646 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:16:17.646 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:16:17.646 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:16:17.647 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-03-25T00:16:17.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '1' '9' '4' '3' '10') 2024-03-25T00:16:17.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-03-25T00:16:17.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2024-03-25T00:16:17.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2024-03-25T00:16:18.291 INFO:tasks.workunit.client.0.smithi080.stderr:osd.10 is already out. 2024-03-25T00:16:18.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-03-25T00:16:23.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-jerasure SOMETHING 2024-03-25T00:16:23.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:16:23.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:16:23.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<10\>' 2024-03-25T00:16:23.310 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-03-25T00:16:23.310 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:16:23.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-03-25T00:16:23.749 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:16:23.749 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:16:23.749 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:16:23.749 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:16:23.749 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:16:23.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 6 2024-03-25T00:16:23.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:16:23.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:16:23.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 10 2024-03-25T00:16:24.327 INFO:tasks.workunit.client.0.smithi080.stderr:osd.10 is already in. 2024-03-25T00:16:24.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-03-25T00:16:24.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:227: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2024-03-25T00:16:24.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:104: rados_osds_out_in: local dir=td/test-erasure-code 2024-03-25T00:16:24.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: local poolname=pool-jerasure 2024-03-25T00:16:24.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: local objname=SOMETHING 2024-03-25T00:16:24.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-03-25T00:16:24.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 FFFF 2024-03-25T00:16:24.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-03-25T00:16:24.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 GGGG 2024-03-25T00:16:24.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-03-25T00:16:24.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 HHHH 2024-03-25T00:16:24.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-03-25T00:16:24.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 IIII 2024-03-25T00:16:24.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:116: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-03-25T00:16:27.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:117: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:16:27.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:118: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:16:27.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:119: rados_osds_out_in: rm td/test-erasure-code/COPY 2024-03-25T00:16:27.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: wait_for_clean 2024-03-25T00:16:27.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:16:27.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:16:27.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:16:27.388 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:16:27.388 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:16:27.388 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:16:27.388 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:16:27.389 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:16:27.389 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:16:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:16:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:16:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:16:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:16:27.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:16:27.479 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:16:27.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:16:27.904 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:16:27.904 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:16:27.904 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:16:27.904 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:16:27.905 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:16:27.905 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:16:27.905 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:16:27.905 INFO:tasks.workunit.client.0.smithi080.stderr:8 2024-03-25T00:16:27.905 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:16:27.905 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:16:27.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:16:27.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:27.905 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:16:28.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836602 2024-03-25T00:16:28.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836602 2024-03-25T00:16:28.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602' 2024-03-25T00:16:28.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:28.245 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:16:28.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673077 2024-03-25T00:16:28.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673077 2024-03-25T00:16:28.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673077' 2024-03-25T00:16:28.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:28.408 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:16:28.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509549 2024-03-25T00:16:28.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509549 2024-03-25T00:16:28.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673077 2-64424509549' 2024-03-25T00:16:28.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:28.691 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:16:28.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481826 2024-03-25T00:16:28.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481826 2024-03-25T00:16:28.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673077 2-64424509549 3-330712481826' 2024-03-25T00:16:28.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:28.846 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:16:28.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182499 2024-03-25T00:16:28.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182499 2024-03-25T00:16:28.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673077 2-64424509549 3-330712481826 4-107374182499' 2024-03-25T00:16:28.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:28.997 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:16:29.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285597 2024-03-25T00:16:29.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285597 2024-03-25T00:16:29.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673077 2-64424509549 3-330712481826 4-107374182499 5-356482285597' 2024-03-25T00:16:29.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:29.149 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:16:29.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696917 2024-03-25T00:16:29.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696917 2024-03-25T00:16:29.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673077 2-64424509549 3-330712481826 4-107374182499 5-356482285597 6-433791696917' 2024-03-25T00:16:29.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:29.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T00:16:29.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691910 2024-03-25T00:16:29.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691910 2024-03-25T00:16:29.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673077 2-64424509549 3-330712481826 4-107374182499 5-356482285597 6-433791696917 7-171798691910' 2024-03-25T00:16:29.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:29.445 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T00:16:29.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528382 2024-03-25T00:16:29.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528382 2024-03-25T00:16:29.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673077 2-64424509549 3-330712481826 4-107374182499 5-356482285597 6-433791696917 7-171798691910 8-193273528382' 2024-03-25T00:16:29.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:29.631 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T00:16:29.770 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467984 2024-03-25T00:16:29.770 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467984 2024-03-25T00:16:29.770 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673077 2-64424509549 3-330712481826 4-107374182499 5-356482285597 6-433791696917 7-171798691910 8-193273528382 9-463856467984' 2024-03-25T00:16:29.770 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:29.770 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-03-25T00:16:29.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201336 2024-03-25T00:16:29.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201336 2024-03-25T00:16:29.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836602 1-42949673077 2-64424509549 3-330712481826 4-107374182499 5-356482285597 6-433791696917 7-171798691910 8-193273528382 9-463856467984 10-236223201336' 2024-03-25T00:16:29.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:29.916 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836602 2024-03-25T00:16:29.916 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:29.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:16:29.918 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836602 2024-03-25T00:16:29.918 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:29.919 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 21474836602 2024-03-25T00:16:29.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836602 2024-03-25T00:16:29.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836602' 2024-03-25T00:16:29.919 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:16:30.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836603 -lt 21474836602 2024-03-25T00:16:30.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:30.323 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:30.323 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673077 2024-03-25T00:16:30.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:16:30.324 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673077 2024-03-25T00:16:30.324 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:30.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673077 2024-03-25T00:16:30.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673077' 2024-03-25T00:16:30.327 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 42949673077 2024-03-25T00:16:30.327 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:16:30.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673077 -lt 42949673077 2024-03-25T00:16:30.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:30.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509549 2024-03-25T00:16:30.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:30.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:16:30.759 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509549 2024-03-25T00:16:30.760 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:30.760 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 64424509549 2024-03-25T00:16:30.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509549 2024-03-25T00:16:30.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509549' 2024-03-25T00:16:30.760 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:16:31.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509550 -lt 64424509549 2024-03-25T00:16:31.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:31.174 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-330712481826 2024-03-25T00:16:31.174 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:31.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:16:31.177 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-330712481826 2024-03-25T00:16:31.177 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:31.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481826 2024-03-25T00:16:31.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 330712481826' 2024-03-25T00:16:31.179 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 330712481826 2024-03-25T00:16:31.179 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:16:31.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481826 -lt 330712481826 2024-03-25T00:16:31.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:31.589 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182499 2024-03-25T00:16:31.589 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:31.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:16:31.591 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182499 2024-03-25T00:16:31.591 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:31.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182499 2024-03-25T00:16:31.592 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 107374182499 2024-03-25T00:16:31.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182499' 2024-03-25T00:16:31.593 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:16:32.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182500 -lt 107374182499 2024-03-25T00:16:32.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:32.004 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285597 2024-03-25T00:16:32.004 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:32.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:16:32.006 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:32.006 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285597 2024-03-25T00:16:32.007 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285597 2024-03-25T00:16:32.007 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285597' 2024-03-25T00:16:32.008 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 356482285597 2024-03-25T00:16:32.008 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:16:32.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285597 -lt 356482285597 2024-03-25T00:16:32.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:32.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:32.403 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-433791696917 2024-03-25T00:16:32.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:16:32.405 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-433791696917 2024-03-25T00:16:32.405 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:32.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696917 2024-03-25T00:16:32.406 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 433791696917 2024-03-25T00:16:32.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 433791696917' 2024-03-25T00:16:32.407 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:16:32.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696918 -lt 433791696917 2024-03-25T00:16:32.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:32.930 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691910 2024-03-25T00:16:32.930 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:32.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T00:16:32.930 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691910 2024-03-25T00:16:32.930 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:32.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691910 2024-03-25T00:16:32.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691910' 2024-03-25T00:16:32.931 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.7 seq 171798691910 2024-03-25T00:16:32.931 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T00:16:33.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691911 -lt 171798691910 2024-03-25T00:16:33.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:33.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528382 2024-03-25T00:16:33.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:33.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T00:16:33.240 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528382 2024-03-25T00:16:33.240 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:33.242 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528382 2024-03-25T00:16:33.242 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528382' 2024-03-25T00:16:33.242 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.8 seq 193273528382 2024-03-25T00:16:33.242 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T00:16:33.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528382 -lt 193273528382 2024-03-25T00:16:33.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:33.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-463856467984 2024-03-25T00:16:33.685 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:33.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T00:16:33.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-463856467984 2024-03-25T00:16:33.687 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:33.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467984 2024-03-25T00:16:33.689 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.9 seq 463856467984 2024-03-25T00:16:33.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 463856467984' 2024-03-25T00:16:33.689 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T00:16:34.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467984 -lt 463856467984 2024-03-25T00:16:34.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:34.131 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:34.131 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201336 2024-03-25T00:16:34.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-03-25T00:16:34.133 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201336 2024-03-25T00:16:34.133 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:34.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201336 2024-03-25T00:16:34.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201336' 2024-03-25T00:16:34.135 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.10 seq 236223201336 2024-03-25T00:16:34.135 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-03-25T00:16:34.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201336 -lt 236223201336 2024-03-25T00:16:34.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:16:34.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:16:34.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:16:35.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-03-25T00:16:35.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:16:35.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:16:35.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:16:35.086 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:16:35.086 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:16:35.086 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:16:35.086 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:16:35.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-03-25T00:16:35.498 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:16:35.498 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:16:35.498 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:16:36.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-03-25T00:16:36.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:16:36.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:16:36.033 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-03-25T00:16:36.034 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:16:36.034 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:16:36.034 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-03-25T00:16:36.034 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: osds=('5' '1' '9' '4' '3' '10') 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: local -a osds 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-03-25T00:16:36.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 5 2024-03-25T00:16:36.947 INFO:tasks.workunit.client.0.smithi080.stderr:osd.5 is already out. 2024-03-25T00:16:36.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-03-25T00:16:36.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 1 2024-03-25T00:16:39.319 INFO:tasks.workunit.client.0.smithi080.stderr:osd.1 is already out. 2024-03-25T00:16:39.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: wait_for_clean 2024-03-25T00:16:39.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:16:39.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:16:39.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:16:39.343 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:16:39.343 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:16:39.343 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:16:39.343 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:16:39.343 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:16:39.343 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:16:39.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:16:39.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:16:39.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:16:39.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:16:39.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:16:39.450 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:16:40.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:16:40.565 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:16:40.565 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:16:40.565 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:16:40.565 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:16:40.565 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:16:40.565 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:16:40.565 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:16:40.565 INFO:tasks.workunit.client.0.smithi080.stderr:8 2024-03-25T00:16:40.566 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:16:40.566 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:16:40.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:16:40.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:40.566 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:16:41.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836607 2024-03-25T00:16:41.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836607 2024-03-25T00:16:41.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607' 2024-03-25T00:16:41.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:41.059 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:16:41.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673081 2024-03-25T00:16:41.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673081 2024-03-25T00:16:41.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081' 2024-03-25T00:16:41.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:41.352 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:16:41.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509554 2024-03-25T00:16:41.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509554 2024-03-25T00:16:41.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509554' 2024-03-25T00:16:41.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:41.512 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:16:41.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481831 2024-03-25T00:16:41.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481831 2024-03-25T00:16:41.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509554 3-330712481831' 2024-03-25T00:16:41.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:41.768 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:16:41.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182504 2024-03-25T00:16:41.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182504 2024-03-25T00:16:41.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509554 3-330712481831 4-107374182504' 2024-03-25T00:16:41.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:41.929 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:16:42.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285601 2024-03-25T00:16:42.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285601 2024-03-25T00:16:42.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509554 3-330712481831 4-107374182504 5-356482285601' 2024-03-25T00:16:42.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:42.379 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:16:42.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696922 2024-03-25T00:16:42.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696922 2024-03-25T00:16:42.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509554 3-330712481831 4-107374182504 5-356482285601 6-433791696922' 2024-03-25T00:16:42.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:42.537 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T00:16:42.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691915 2024-03-25T00:16:42.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691915 2024-03-25T00:16:42.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509554 3-330712481831 4-107374182504 5-356482285601 6-433791696922 7-171798691915' 2024-03-25T00:16:42.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:42.690 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T00:16:42.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528386 2024-03-25T00:16:42.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528386 2024-03-25T00:16:42.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509554 3-330712481831 4-107374182504 5-356482285601 6-433791696922 7-171798691915 8-193273528386' 2024-03-25T00:16:42.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:42.844 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T00:16:42.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467988 2024-03-25T00:16:42.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467988 2024-03-25T00:16:42.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509554 3-330712481831 4-107374182504 5-356482285601 6-433791696922 7-171798691915 8-193273528386 9-463856467988' 2024-03-25T00:16:42.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:42.997 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-03-25T00:16:43.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201340 2024-03-25T00:16:43.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201340 2024-03-25T00:16:43.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673081 2-64424509554 3-330712481831 4-107374182504 5-356482285601 6-433791696922 7-171798691915 8-193273528386 9-463856467988 10-236223201340' 2024-03-25T00:16:43.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:43.155 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836607 2024-03-25T00:16:43.155 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:43.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:16:43.157 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836607 2024-03-25T00:16:43.157 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:43.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836607 2024-03-25T00:16:43.158 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 21474836607 2024-03-25T00:16:43.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2024-03-25T00:16:43.159 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:16:43.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836607 -lt 21474836607 2024-03-25T00:16:43.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:43.554 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673081 2024-03-25T00:16:43.554 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:43.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:16:43.556 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673081 2024-03-25T00:16:43.557 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:43.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673081 2024-03-25T00:16:43.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673081' 2024-03-25T00:16:43.558 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 42949673081 2024-03-25T00:16:43.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:16:44.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673082 -lt 42949673081 2024-03-25T00:16:44.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:44.054 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509554 2024-03-25T00:16:44.054 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:44.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:16:44.057 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509554 2024-03-25T00:16:44.057 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:44.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509554 2024-03-25T00:16:44.058 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 64424509554 2024-03-25T00:16:44.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509554' 2024-03-25T00:16:44.059 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:16:44.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509554 -lt 64424509554 2024-03-25T00:16:44.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:44.510 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-330712481831 2024-03-25T00:16:44.510 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:44.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:16:44.511 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-330712481831 2024-03-25T00:16:44.511 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:44.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481831 2024-03-25T00:16:44.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 330712481831' 2024-03-25T00:16:44.513 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 330712481831 2024-03-25T00:16:44.514 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:16:44.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481831 -lt 330712481831 2024-03-25T00:16:44.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:44.926 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182504 2024-03-25T00:16:44.926 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:44.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:16:44.928 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182504 2024-03-25T00:16:44.928 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:44.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182504 2024-03-25T00:16:44.929 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 107374182504 2024-03-25T00:16:44.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182504' 2024-03-25T00:16:44.930 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:16:45.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182505 -lt 107374182504 2024-03-25T00:16:45.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:45.366 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285601 2024-03-25T00:16:45.367 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:45.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:16:45.368 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285601 2024-03-25T00:16:45.368 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:45.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285601 2024-03-25T00:16:45.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285601' 2024-03-25T00:16:45.370 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 356482285601 2024-03-25T00:16:45.370 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:16:45.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285602 -lt 356482285601 2024-03-25T00:16:45.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:45.818 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-433791696922 2024-03-25T00:16:45.818 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:45.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:16:45.820 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:45.820 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-433791696922 2024-03-25T00:16:45.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696922 2024-03-25T00:16:45.821 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 433791696922 2024-03-25T00:16:45.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 433791696922' 2024-03-25T00:16:45.822 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:16:46.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696923 -lt 433791696922 2024-03-25T00:16:46.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:46.245 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691915 2024-03-25T00:16:46.245 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:46.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T00:16:46.247 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691915 2024-03-25T00:16:46.248 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:46.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691915 2024-03-25T00:16:46.249 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.7 seq 171798691915 2024-03-25T00:16:46.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691915' 2024-03-25T00:16:46.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T00:16:46.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691915 -lt 171798691915 2024-03-25T00:16:46.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:46.649 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528386 2024-03-25T00:16:46.649 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:46.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T00:16:46.652 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528386 2024-03-25T00:16:46.652 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:46.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528386 2024-03-25T00:16:46.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528386' 2024-03-25T00:16:46.653 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.8 seq 193273528386 2024-03-25T00:16:46.654 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T00:16:47.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528387 -lt 193273528386 2024-03-25T00:16:47.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:47.061 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-463856467988 2024-03-25T00:16:47.061 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:47.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T00:16:47.065 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-463856467988 2024-03-25T00:16:47.065 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:47.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467988 2024-03-25T00:16:47.065 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.9 seq 463856467988 2024-03-25T00:16:47.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 463856467988' 2024-03-25T00:16:47.066 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T00:16:47.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467989 -lt 463856467988 2024-03-25T00:16:47.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:47.462 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201340 2024-03-25T00:16:47.463 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:47.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-03-25T00:16:47.465 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201340 2024-03-25T00:16:47.465 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:47.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201340 2024-03-25T00:16:47.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201340' 2024-03-25T00:16:47.467 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.10 seq 236223201340 2024-03-25T00:16:47.467 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-03-25T00:16:47.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201341 -lt 236223201340 2024-03-25T00:16:47.877 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:16:47.877 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:16:47.878 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:16:48.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-03-25T00:16:48.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:16:48.410 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:16:48.410 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:16:48.410 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:16:48.410 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:16:48.410 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:16:48.410 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:16:48.801 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-03-25T00:16:48.802 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:16:48.802 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:16:48.802 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:16:49.343 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-03-25T00:16:49.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:16:49.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:16:49.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-03-25T00:16:49.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-03-25T00:16:49.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:16:49.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:16:49.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<5\>' 2024-03-25T00:16:49.345 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:16:49.345 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-03-25T00:16:49.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-25T00:16:49.750 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:16:49.750 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:16:49.750 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:16:49.750 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:16:49.750 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:16:49.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-03-25T00:16:49.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-03-25T00:16:49.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-03-25T00:16:49.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:16:49.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:16:49.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<1\>' 2024-03-25T00:16:49.751 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-03-25T00:16:49.751 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:16:50.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-25T00:16:50.206 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:16:50.206 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:16:50.207 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:16:50.207 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:16:50.207 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:16:50.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-03-25T00:16:50.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:140: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:16:50.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:141: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:16:50.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-03-25T00:16:50.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 5 2024-03-25T00:16:50.806 INFO:tasks.workunit.client.0.smithi080.stderr:osd.5 is already in. 2024-03-25T00:16:50.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-03-25T00:16:50.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 1 2024-03-25T00:16:52.908 INFO:tasks.workunit.client.0.smithi080.stderr:osd.1 is already in. 2024-03-25T00:16:52.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:150: rados_osds_out_in: wait_for_clean 2024-03-25T00:16:52.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:16:52.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:16:52.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:16:52.932 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:16:52.933 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:16:52.933 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:16:52.933 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:16:52.933 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:16:52.933 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:16:53.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:16:53.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:16:53.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:16:53.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:16:53.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:16:53.043 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:16:53.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:16:53.905 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:16:53.906 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:16:53.906 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:16:53.906 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:16:53.906 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:16:53.906 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:16:53.906 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:16:53.907 INFO:tasks.workunit.client.0.smithi080.stderr:8 2024-03-25T00:16:53.907 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:16:53.907 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:16:53.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:16:53.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:53.908 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:16:54.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836612 2024-03-25T00:16:54.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836612 2024-03-25T00:16:54.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612' 2024-03-25T00:16:54.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:54.062 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:16:54.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673086 2024-03-25T00:16:54.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673086 2024-03-25T00:16:54.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673086' 2024-03-25T00:16:54.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:54.450 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:16:54.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509558 2024-03-25T00:16:54.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509558 2024-03-25T00:16:54.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673086 2-64424509558' 2024-03-25T00:16:54.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:54.611 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:16:55.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481835 2024-03-25T00:16:55.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481835 2024-03-25T00:16:55.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673086 2-64424509558 3-330712481835' 2024-03-25T00:16:55.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:55.150 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:16:55.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182509 2024-03-25T00:16:55.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182509 2024-03-25T00:16:55.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673086 2-64424509558 3-330712481835 4-107374182509' 2024-03-25T00:16:55.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:55.556 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:16:55.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285606 2024-03-25T00:16:55.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285606 2024-03-25T00:16:55.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673086 2-64424509558 3-330712481835 4-107374182509 5-356482285606' 2024-03-25T00:16:55.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:55.824 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:16:56.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696927 2024-03-25T00:16:56.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696927 2024-03-25T00:16:56.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673086 2-64424509558 3-330712481835 4-107374182509 5-356482285606 6-433791696927' 2024-03-25T00:16:56.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:56.267 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T00:16:56.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691919 2024-03-25T00:16:56.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691919 2024-03-25T00:16:56.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673086 2-64424509558 3-330712481835 4-107374182509 5-356482285606 6-433791696927 7-171798691919' 2024-03-25T00:16:56.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:56.421 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T00:16:56.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528391 2024-03-25T00:16:56.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528391 2024-03-25T00:16:56.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673086 2-64424509558 3-330712481835 4-107374182509 5-356482285606 6-433791696927 7-171798691919 8-193273528391' 2024-03-25T00:16:56.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:56.575 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-25T00:16:56.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467993 2024-03-25T00:16:56.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467993 2024-03-25T00:16:56.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673086 2-64424509558 3-330712481835 4-107374182509 5-356482285606 6-433791696927 7-171798691919 8-193273528391 9-463856467993' 2024-03-25T00:16:56.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:16:56.729 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-03-25T00:16:56.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201345 2024-03-25T00:16:56.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201345 2024-03-25T00:16:56.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836612 1-42949673086 2-64424509558 3-330712481835 4-107374182509 5-356482285606 6-433791696927 7-171798691919 8-193273528391 9-463856467993 10-236223201345' 2024-03-25T00:16:56.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:56.880 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836612 2024-03-25T00:16:56.880 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:56.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:16:56.882 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836612 2024-03-25T00:16:56.882 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:56.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836612 2024-03-25T00:16:56.883 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 21474836612 2024-03-25T00:16:56.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836612' 2024-03-25T00:16:56.884 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:16:57.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836612 -lt 21474836612 2024-03-25T00:16:57.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:57.443 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673086 2024-03-25T00:16:57.443 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:57.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:16:57.446 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673086 2024-03-25T00:16:57.446 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:57.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673086 2024-03-25T00:16:57.447 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 42949673086 2024-03-25T00:16:57.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673086' 2024-03-25T00:16:57.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:16:58.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673086 -lt 42949673086 2024-03-25T00:16:58.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:58.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509558 2024-03-25T00:16:58.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:58.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:16:58.087 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509558 2024-03-25T00:16:58.088 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:58.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509558 2024-03-25T00:16:58.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509558' 2024-03-25T00:16:58.090 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 64424509558 2024-03-25T00:16:58.090 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:16:58.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509559 -lt 64424509558 2024-03-25T00:16:58.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:58.564 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-330712481835 2024-03-25T00:16:58.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:58.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:16:58.567 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-330712481835 2024-03-25T00:16:58.567 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:58.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481835 2024-03-25T00:16:58.568 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 330712481835 2024-03-25T00:16:58.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 330712481835' 2024-03-25T00:16:58.569 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:16:59.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481836 -lt 330712481835 2024-03-25T00:16:59.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:59.218 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182509 2024-03-25T00:16:59.218 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:59.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:16:59.220 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182509 2024-03-25T00:16:59.221 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:59.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182509 2024-03-25T00:16:59.222 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 107374182509 2024-03-25T00:16:59.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182509' 2024-03-25T00:16:59.222 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:16:59.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182509 -lt 107374182509 2024-03-25T00:16:59.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:16:59.678 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285606 2024-03-25T00:16:59.678 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:16:59.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:16:59.680 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285606 2024-03-25T00:16:59.680 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:16:59.681 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 356482285606 2024-03-25T00:16:59.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285606 2024-03-25T00:16:59.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285606' 2024-03-25T00:16:59.682 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:17:00.074 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285607 -lt 356482285606 2024-03-25T00:17:00.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:17:00.075 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-433791696927 2024-03-25T00:17:00.075 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:17:00.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:17:00.078 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-433791696927 2024-03-25T00:17:00.078 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:17:00.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696927 2024-03-25T00:17:00.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 433791696927' 2024-03-25T00:17:00.079 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 433791696927 2024-03-25T00:17:00.080 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:17:00.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696927 -lt 433791696927 2024-03-25T00:17:00.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:17:00.554 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691919 2024-03-25T00:17:00.555 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:17:00.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T00:17:00.557 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691919 2024-03-25T00:17:00.557 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:17:00.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691919 2024-03-25T00:17:00.558 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.7 seq 171798691919 2024-03-25T00:17:00.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691919' 2024-03-25T00:17:00.559 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T00:17:01.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691920 -lt 171798691919 2024-03-25T00:17:01.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:17:01.037 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528391 2024-03-25T00:17:01.037 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:17:01.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T00:17:01.039 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528391 2024-03-25T00:17:01.039 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:17:01.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528391 2024-03-25T00:17:01.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528391' 2024-03-25T00:17:01.042 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.8 seq 193273528391 2024-03-25T00:17:01.042 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T00:17:01.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528392 -lt 193273528391 2024-03-25T00:17:01.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:17:01.449 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-463856467993 2024-03-25T00:17:01.449 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:17:01.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-25T00:17:01.451 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-463856467993 2024-03-25T00:17:01.451 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:17:01.452 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.9 seq 463856467993 2024-03-25T00:17:01.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467993 2024-03-25T00:17:01.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 463856467993' 2024-03-25T00:17:01.453 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-25T00:17:01.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467994 -lt 463856467993 2024-03-25T00:17:01.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:17:01.858 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:17:01.858 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201345 2024-03-25T00:17:01.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-03-25T00:17:01.860 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201345 2024-03-25T00:17:01.860 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:17:01.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201345 2024-03-25T00:17:01.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201345' 2024-03-25T00:17:01.862 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.10 seq 236223201345 2024-03-25T00:17:01.862 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-03-25T00:17:02.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201346 -lt 236223201345 2024-03-25T00:17:02.281 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:17:02.281 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:17:02.281 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:17:02.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-03-25T00:17:02.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:17:02.822 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:17:02.822 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:17:02.822 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:17:02.823 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:17:02.823 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:17:02.823 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:17:03.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-03-25T00:17:03.227 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:17:03.228 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:17:03.228 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:17:03.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-03-25T00:17:03.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:17:03.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:17:03.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-03-25T00:17:03.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:17:03.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:17:03.755 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-03-25T00:17:03.755 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:17:04.176 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-03-25T00:17:04.177 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:17:04.177 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:17:04.177 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:17:04.177 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:17:04.177 INFO:tasks.workunit.client.0.smithi080.stderr:10' 2024-03-25T00:17:04.177 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-03-25T00:17:04.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2024-03-25T00:17:04.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2024-03-25T00:17:04.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:229: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2024-03-25T00:17:04.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-jerasure 2024-03-25T00:17:04.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-03-25T00:17:04.676 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' does not exist 2024-03-25T00:17:04.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2024-03-25T00:17:05.377 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile profile-jerasure does not exist 2024-03-25T00:17:05.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-03-25T00:17:05.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2024-03-25T00:17:05.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2024-03-25T00:17:05.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:157: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2024-03-25T00:17:05.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:158: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2024-03-25T00:17:05.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:160: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2024-03-25T00:17:06.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:165: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2024-03-25T00:17:06.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2024-03-25T00:17:08.399 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-lrc-a' already exists 2024-03-25T00:17:08.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:17:09.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:168: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2024-03-25T00:17:09.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-03-25T00:17:09.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2024-03-25T00:17:09.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-03-25T00:17:09.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:09.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-03-25T00:17:09.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:09.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-03-25T00:17:09.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:09.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-03-25T00:17:09.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:09.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-03-25T00:17:09.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2024-03-25T00:17:10.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:17:10.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:17:10.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-03-25T00:17:10.356 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-03-25T00:17:10.356 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-03-25T00:17:10.356 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:17:10.357 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-03-25T00:17:10.357 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:17:11.008 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-03-25T00:17:11.008 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:17:11.009 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T00:17:11.009 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 3 2024-03-25T00:17:11.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '9' '3') 2024-03-25T00:17:11.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-03-25T00:17:11.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-03-25T00:17:11.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2024-03-25T00:17:12.100 INFO:tasks.workunit.client.0.smithi080.stderr:osd.3 is already out. 2024-03-25T00:17:12.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-03-25T00:17:17.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-03-25T00:17:17.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-03-25T00:17:17.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:17:17.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<3\>' 2024-03-25T00:17:17.126 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-03-25T00:17:17.127 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:17:17.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-03-25T00:17:17.555 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:17:17.555 INFO:tasks.workunit.client.0.smithi080.stderr:7' 2024-03-25T00:17:17.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 7 2024-03-25T00:17:17.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:17:17.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:17:17.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 3 2024-03-25T00:17:18.197 INFO:tasks.workunit.client.0.smithi080.stderr:osd.3 is already in. 2024-03-25T00:17:18.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-03-25T00:17:18.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:170: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2024-03-25T00:17:18.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc-a 2024-03-25T00:17:18.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2024-03-25T00:17:19.476 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-lrc-a' does not exist 2024-03-25T00:17:19.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2024-03-25T00:17:21.097 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile profile-lrc-a does not exist 2024-03-25T00:17:21.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-03-25T00:17:21.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2024-03-25T00:17:21.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2024-03-25T00:17:21.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:176: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2024-03-25T00:17:21.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:177: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2024-03-25T00:17:21.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2024-03-25T00:17:21.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:183: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2024-03-25T00:17:21.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2024-03-25T00:17:22.967 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-lrc' already exists 2024-03-25T00:17:22.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:17:23.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:186: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2024-03-25T00:17:23.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-03-25T00:17:23.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2024-03-25T00:17:23.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-03-25T00:17:23.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:23.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-03-25T00:17:23.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:23.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-03-25T00:17:23.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:23.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-03-25T00:17:23.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:23.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-03-25T00:17:23.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2024-03-25T00:17:25.007 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:17:25.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:17:25.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-03-25T00:17:25.350 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2024-03-25T00:17:25.350 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-03-25T00:17:25.350 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:17:25.350 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-03-25T00:17:25.350 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:17:25.853 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:10 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:7' 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 7 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('8' '6' '0' '10' '5' '3' '9' '7') 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2024-03-25T00:17:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2024-03-25T00:17:27.267 INFO:tasks.workunit.client.0.smithi080.stderr:osd.7 is already out. 2024-03-25T00:17:27.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-03-25T00:17:32.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc SOMETHING 2024-03-25T00:17:32.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-03-25T00:17:32.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:17:32.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2024-03-25T00:17:32.295 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-03-25T00:17:32.295 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:17:32.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-03-25T00:17:32.725 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:17:32.725 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:17:32.725 INFO:tasks.workunit.client.0.smithi080.stderr:10 2024-03-25T00:17:32.725 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:17:32.725 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:17:32.725 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:17:32.725 INFO:tasks.workunit.client.0.smithi080.stderr:4' 2024-03-25T00:17:32.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 4 2024-03-25T00:17:32.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:17:32.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:17:32.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2024-03-25T00:17:34.100 INFO:tasks.workunit.client.0.smithi080.stderr:osd.7 is already in. 2024-03-25T00:17:34.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-03-25T00:17:34.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:188: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2024-03-25T00:17:34.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc 2024-03-25T00:17:34.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2024-03-25T00:17:36.074 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-lrc' does not exist 2024-03-25T00:17:36.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2024-03-25T00:17:37.432 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile profile-lrc does not exist 2024-03-25T00:17:37.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-03-25T00:17:37.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2024-03-25T00:17:37.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:234: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2024-03-25T00:17:37.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:236: TEST_rados_put_get_shec: local poolname=pool-shec 2024-03-25T00:17:37.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:237: TEST_rados_put_get_shec: local profile=profile-shec 2024-03-25T00:17:37.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2024-03-25T00:17:38.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:243: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2024-03-25T00:17:38.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2024-03-25T00:17:40.129 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-shec' already exists 2024-03-25T00:17:40.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:17:41.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:246: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2024-03-25T00:17:41.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-03-25T00:17:41.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2024-03-25T00:17:41.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-03-25T00:17:41.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:41.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-03-25T00:17:41.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:41.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-03-25T00:17:41.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:41.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-03-25T00:17:41.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-03-25T00:17:41.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-03-25T00:17:41.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2024-03-25T00:17:43.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:17:43.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:17:43.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-03-25T00:17:43.393 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2024-03-25T00:17:43.393 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-03-25T00:17:43.393 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:17:43.394 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-03-25T00:17:43.394 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:17:43.937 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-03-25T00:17:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:17:43.937 INFO:tasks.workunit.client.0.smithi080.stderr:1' 2024-03-25T00:17:43.938 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 1 2024-03-25T00:17:43.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('7' '9' '1') 2024-03-25T00:17:43.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-03-25T00:17:43.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-03-25T00:17:43.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2024-03-25T00:17:44.918 INFO:tasks.workunit.client.0.smithi080.stderr:osd.1 is already out. 2024-03-25T00:17:44.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-03-25T00:17:49.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-shec SOMETHING 2024-03-25T00:17:49.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-03-25T00:17:49.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-25T00:17:49.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<1\>' 2024-03-25T00:17:49.945 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-03-25T00:17:49.945 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:17:50.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-03-25T00:17:50.423 INFO:tasks.workunit.client.0.smithi080.stderr:9 2024-03-25T00:17:50.423 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:17:50.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 6 2024-03-25T00:17:50.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-03-25T00:17:50.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-03-25T00:17:50.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 1 2024-03-25T00:17:51.070 INFO:tasks.workunit.client.0.smithi080.stderr:osd.1 is already in. 2024-03-25T00:17:51.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-03-25T00:17:51.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:248: TEST_rados_put_get_shec: delete_pool pool-shec 2024-03-25T00:17:51.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-shec 2024-03-25T00:17:51.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2024-03-25T00:17:52.699 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-shec' does not exist 2024-03-25T00:17:52.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2024-03-25T00:17:54.454 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile profile-shec does not exist 2024-03-25T00:17:54.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2024-03-25T00:17:54.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=ecpool 2024-03-25T00:17:54.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2024-03-25T00:17:55.141 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'ecpool' does not exist 2024-03-25T00:17:55.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2024-03-25T00:17:55.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-03-25T00:17:55.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:17:55.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-03-25T00:17:55.160 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:17:55.160 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:17:55.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:17:55.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:17:55.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:17:55.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:17:55.479 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:17:55.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:17:55.481 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:17:55.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:17:55.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:17:55.482 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:17:55.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:17:55.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:17:55.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:17:55.484 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:17:55.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:17:55.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:17:55.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-03-25T00:17:55.594 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:17:55.594 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:17:55.594 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:17:55.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48948 2024-03-25T00:17:55.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:17:55.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-code 0 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:17:55.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:17:55.601 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:17:55.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:17:55.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:17:55.601 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:17:55.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:17:55.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:17:55.603 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:17:55.603 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:17:55.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:17:55.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-25T00:17:55.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-03-25T00:17:55.606 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:17:55.607 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:17:55.607 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48948 2024-03-25T00:17:55.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48948 2024-03-25T00:17:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:17:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:17:55.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-25T00:17:55.609 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-25T00:17:55.610 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-25T00:17:55.622 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2024-03-25T00:17:55.622 DEBUG:teuthology.orchestra.run.smithi080:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9c429fb6abfb5deed62697d56d8997d0f0d6d83f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2024-03-25T00:17:55.676 INFO:tasks.workunit.client.0.smithi080.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-25T00:17:55.682 INFO:tasks.workunit.client.0.smithi080.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-25T00:17:55.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T00:17:55.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-03-25T00:17:55.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-25T00:17:55.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-25T00:17:55.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-25T00:17:55.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-25T00:17:55.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-25T00:17:55.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-25T00:17:55.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-eio 2024-03-25T00:17:55.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2024-03-25T00:17:55.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2024-03-25T00:17:55.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2024-03-25T00:17:55.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2024-03-25T00:17:55.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2024-03-25T00:17:55.683 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2024-03-25T00:17:55.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none ' 2024-03-25T00:17:55.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2024-03-25T00:17:55.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-03-25T00:17:55.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2024-03-25T00:17:55.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_ec_object_attr_read_error 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_ec_recovery_multiple_errors 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_ec_recovery_multiple_objects 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_ec_recovery_multiple_objects_eio 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_ec_recovery_unfound 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_ec_single_recovery_error 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_get_bad_size_shard_0 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_get_bad_size_shard_1 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_get_subread_eio_shard_0 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_get_subread_eio_shard_1 2024-03-25T00:17:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_get_subread_missing 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:17:55.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:17:55.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:17:55.689 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:17:55.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:17:55.690 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:17:55.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:17:55.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:17:55.691 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:17:55.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:17:55.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:17:55.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:17:55.692 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:17:55.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:17:55.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:17:55.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:17:55.695 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:17:55.696 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:17:55.696 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:17:55.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:17:55.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:17:55.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:17:55.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-03-25T00:17:55.697 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:17:55.698 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:17:55.698 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:17:55.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81310 2024-03-25T00:17:55.699 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:17:55.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-25T00:17:55.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-25T00:17:55.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:17:55.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-03-25T00:17:55.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-03-25T00:17:55.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-03-25T00:17:55.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:17:55.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:17:55.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:17:55.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-03-25T00:17:55.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-03-25T00:17:56.176 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:17:56.176 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:17:56.176 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:17:56.177 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:17:56.177 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:17:56.177 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:17:56.177 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:17:56.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T00:17:56.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:17:56.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:17:56.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:17:56.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:17:56.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:17:56.381 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:17:56.381 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:17:56.381 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:17:56.381 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:17:56.382 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:17:56.382 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:17:56.382 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:17:56.382 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:17:56.382 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:17:56.382 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get fsid 2024-03-25T00:17:56.498 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:17:56.498 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:17:56.498 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:17:56.498 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:17:56.498 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:17:56.499 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:17:56.499 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:17:56.499 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:17:56.500 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:17:56.500 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:17:56.500 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:17:56.500 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:17:56.500 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:17:56.500 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get mon_host 2024-03-25T00:17:56.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-03-25T00:17:56.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-03-25T00:17:56.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:17:56.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:17:56.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:17:56.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-03-25T00:17:56.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:17:57.510 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:17:57.510 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:17:57.510 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:17:57.511 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:17:57.511 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:17:57.511 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:17:57.511 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:17:57.511 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:17:57.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:17:57.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-03-25T00:17:57.548 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T00:17:57.832 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' already exists 2024-03-25T00:17:57.848 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:17:58.851 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-03-25T00:17:58.851 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:17:58.851 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:17:58.851 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:17:58.851 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:17:58.851 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:17:58.852 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:17:58.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-03-25T00:17:58.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-mon.a.asok log flush 2024-03-25T00:17:58.958 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:17:58.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-03-25T00:17:58.973 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:17:56.228+0000 7ff50381dc80 0 load: jerasure load: lrc 2024-03-25T00:17:58.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2024-03-25T00:17:58.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2024-03-25T00:17:58.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_backfill_unfound: local objname=myobject 2024-03-25T00:17:58.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_backfill_unfound: local lastobj=300 2024-03-25T00:17:58.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:515: TEST_ec_backfill_unfound: local testobj=obj250 2024-03-25T00:17:58.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:17:58.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:518: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:17:58.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:519: TEST_ec_backfill_unfound: setup_osds 5 2024-03-25T00:17:58.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-03-25T00:17:58.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-03-25T00:17:58.975 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-03-25T00:17:58.976 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-03-25T00:17:58.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:17:58.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-03-25T00:17:58.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:17:58.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:17:58.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:17:58.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:17:58.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-03-25T00:17:58.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:17:58.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:17:58.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:17:58.981 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-03-25T00:17:58.982 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:17:58.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=67b2b067-7045-48f7-98f9-b3d5c5e99e5f 2024-03-25T00:17:58.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 67b2b067-7045-48f7-98f9-b3d5c5e99e5f' 2024-03-25T00:17:58.983 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 67b2b067-7045-48f7-98f9-b3d5c5e99e5f 2024-03-25T00:17:58.983 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:17:59.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3wgBmqMe0ABAAXkaNuh6q1FM9RH+HbXwBoQ== 2024-03-25T00:17:59.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3wgBmqMe0ABAAXkaNuh6q1FM9RH+HbXwBoQ=="}' 2024-03-25T00:17:59.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 67b2b067-7045-48f7-98f9-b3d5c5e99e5f -i td/test-erasure-eio/0/new.json 2024-03-25T00:17:59.274 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:17:59.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-03-25T00:17:59.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3wgBmqMe0ABAAXkaNuh6q1FM9RH+HbXwBoQ== --osd-uuid 67b2b067-7045-48f7-98f9-b3d5c5e99e5f 2024-03-25T00:17:59.328 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:17:59.336+0000 7fbf6ef7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:17:59.334 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:17:59.344+0000 7fbf6ef7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:17:59.335 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:17:59.344+0000 7fbf6ef7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:17:59.336 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:17:59.344+0000 7fbf6ef7c7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:17:59.336 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:17:59.344+0000 7fbf6ef7c7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:17:59.336 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:17:59.344+0000 7fbf6ef7c7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:17:59.375 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:17:59.384+0000 7fbf6ef7c7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-03-25T00:18:05.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-03-25T00:18:05.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:18:05.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:18:05.313 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:18:05.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:18:05.866 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:18:05.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:18:05.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:18:05.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:18:05.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:18:05.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:18:05.903 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:05.912+0000 7f119fad17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:18:05.949 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:05.960+0000 7f119fad17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:18:05.950 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:05.960+0000 7f119fad17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:18:06.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:18:06.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:18:06.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:18:06.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:18:06.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:18:06.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:06.328 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:18:06.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:18:06.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:06.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:06.514 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:06.524+0000 7f119fad17c0 -1 Falling back to public interface 2024-03-25T00:18:06.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:07.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:07.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:07.789 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:18:07.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:18:07.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:07.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:08.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:09.056 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:09.063+0000 7f119fad17c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:18:09.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:09.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:09.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:18:09.247 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:18:09.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:09.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:09.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:10.776 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:10.776 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:10.776 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:18:10.777 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:18:10.777 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:10.777 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:11.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:12.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:12.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:12.215 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:18:12.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:18:12.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:12.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:12.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:13.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:13.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:13.640 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:18:13.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:18:13.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:13.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:14.135 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:15.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:15.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:15.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:18:15.138 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:18:15.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:15.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:15.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:16.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:16.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:16.595 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:18:16.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:18:16.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:16.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:17.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:18.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:18.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:18.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:18:18.121 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:18:18.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:18.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:18.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:19.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:19.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:19.554 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:18:19.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:18:19.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:19.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:20.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:21.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:21.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:21.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:18:21.004 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:18:21.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:21.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:21.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:22.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:22.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:22.515 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:18:22.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:18:22.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:22.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:23.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:24.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:24.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:24.558 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:18:24.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:18:24.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:24.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:25.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:26.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:26.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:26.054 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:18:26.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:18:26.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:26.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:26.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:27.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:27.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:27.678 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:18:27.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:18:27.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:27.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:28.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:29.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:29.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:29.426 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:18:29.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:18:29.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:29.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:30.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:31.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:31.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:31.239 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:18:31.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:18:31.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:31.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:31.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:32.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:32.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:32.742 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:18:32.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:18:32.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:32.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:18:33.259 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2795353942,v1:127.0.0.1:6803/2795353942] [v2:127.0.0.1:6804/2795353942,v1:127.0.0.1:6805/2795353942] exists,up 67b2b067-7045-48f7-98f9-b3d5c5e99e5f 2024-03-25T00:18:33.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:18:33.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:18:33.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:18:33.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:18:33.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-03-25T00:18:33.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:18:33.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:18:33.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:18:33.263 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:18:33.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:18:33.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:18:33.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:18:33.265 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:18:33.266 INFO:tasks.workunit.client.0.smithi080.stdout:add osd1 df601ba8-b889-46c1-853c-a174827666e8 2024-03-25T00:18:33.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=df601ba8-b889-46c1-853c-a174827666e8 2024-03-25T00:18:33.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 df601ba8-b889-46c1-853c-a174827666e8' 2024-03-25T00:18:33.266 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:18:33.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZwgBmPjfzERAA/pzO4mwSJis5HVKX8lymJQ== 2024-03-25T00:18:33.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZwgBmPjfzERAA/pzO4mwSJis5HVKX8lymJQ=="}' 2024-03-25T00:18:33.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new df601ba8-b889-46c1-853c-a174827666e8 -i td/test-erasure-eio/1/new.json 2024-03-25T00:18:33.822 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:18:33.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-03-25T00:18:33.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZwgBmPjfzERAA/pzO4mwSJis5HVKX8lymJQ== --osd-uuid df601ba8-b889-46c1-853c-a174827666e8 2024-03-25T00:18:33.872 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:33.879+0000 7f3897d6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:18:33.874 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:33.883+0000 7f3897d6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:18:33.875 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:33.883+0000 7f3897d6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:18:33.875 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:33.883+0000 7f3897d6c7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:18:33.876 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:33.883+0000 7f3897d6c7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:18:33.876 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:33.883+0000 7f3897d6c7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:18:33.920 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:33.927+0000 7f3897d6c7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-03-25T00:18:39.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-03-25T00:18:39.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:18:39.982 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd1 key to auth repository 2024-03-25T00:18:39.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T00:18:39.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:18:40.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T00:18:40.577 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.1 2024-03-25T00:18:40.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:18:40.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:18:40.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:18:40.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:18:40.611 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:40.619+0000 7f0cf60087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:18:40.634 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:40.643+0000 7f0cf60087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:18:40.636 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:40.643+0000 7f0cf60087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:18:41.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T00:18:41.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:18:41.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:18:41.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:18:41.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:18:41.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:41.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:18:41.013 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:18:41.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:41.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:41.214 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:41.223+0000 7f0cf60087c0 -1 Falling back to public interface 2024-03-25T00:18:41.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:42.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:42.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:42.426 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:18:42.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:18:42.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:42.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:42.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:43.709 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:18:43.719+0000 7f0cf60087c0 -1 osd.1 0 log_to_monitors true 2024-03-25T00:18:43.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:43.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:43.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:18:43.884 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:18:43.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:43.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:44.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:45.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:45.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:45.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:18:45.500 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:18:45.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:45.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:45.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:46.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:46.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:46.918 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:18:46.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:18:46.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:46.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:47.398 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:48.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:48.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:48.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:18:48.400 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:18:48.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:48.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:49.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:50.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:50.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:50.034 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:18:50.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:18:50.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:50.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:50.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:51.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:51.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:51.545 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:18:51.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:18:51.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:51.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:51.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:53.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:53.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:53.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:18:53.001 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:18:53.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:53.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:53.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:54.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:54.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:54.599 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:18:54.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:18:54.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:54.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:55.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:56.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:56.034 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:18:56.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:56.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:18:56.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:56.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:56.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:57.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:57.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:57.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:18:57.784 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:18:57.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:57.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:18:58.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:18:59.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:18:59.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:18:59.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:18:59.299 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:18:59.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:18:59.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:19:00.131 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:01.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:01.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:01.133 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:19:01.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:19:01.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:01.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:19:01.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:02.809 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:02.809 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:02.809 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:19:02.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:19:02.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:02.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:19:03.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:04.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:04.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:04.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:19:04.959 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:19:04.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:04.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:19:05.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:06.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:06.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:06.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:19:06.624 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:19:06.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:06.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:19:07.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:08.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:08.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:08.189 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:19:08.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:19:08.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:08.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:19:08.656 INFO:tasks.workunit.client.0.smithi080.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2446484528,v1:127.0.0.1:6811/2446484528] [v2:127.0.0.1:6812/2446484528,v1:127.0.0.1:6813/2446484528] exists,up df601ba8-b889-46c1-853c-a174827666e8 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:19:08.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:19:08.661 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:19:08.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:19:08.663 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:19:08.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4298834f-324a-49e0-9eb1-e7d2c17bcdfa 2024-03-25T00:19:08.664 INFO:tasks.workunit.client.0.smithi080.stdout:add osd2 4298834f-324a-49e0-9eb1-e7d2c17bcdfa 2024-03-25T00:19:08.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4298834f-324a-49e0-9eb1-e7d2c17bcdfa' 2024-03-25T00:19:08.664 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:19:08.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8wgBmZuakKRAASJzguiCtRAlEuP+AdZbdog== 2024-03-25T00:19:08.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8wgBmZuakKRAASJzguiCtRAlEuP+AdZbdog=="}' 2024-03-25T00:19:08.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4298834f-324a-49e0-9eb1-e7d2c17bcdfa -i td/test-erasure-eio/2/new.json 2024-03-25T00:19:09.166 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:19:09.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-03-25T00:19:09.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8wgBmZuakKRAASJzguiCtRAlEuP+AdZbdog== --osd-uuid 4298834f-324a-49e0-9eb1-e7d2c17bcdfa 2024-03-25T00:19:09.221 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:09.231+0000 7f93354247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:09.223 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:09.231+0000 7f93354247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:09.224 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:09.235+0000 7f93354247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:09.225 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:09.235+0000 7f93354247c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:19:09.225 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:09.235+0000 7f93354247c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:19:09.225 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:09.235+0000 7f93354247c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:19:09.318 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:09.327+0000 7f93354247c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-03-25T00:19:15.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-03-25T00:19:15.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:19:15.298 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd2 key to auth repository 2024-03-25T00:19:15.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T00:19:15.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:19:15.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T00:19:15.853 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:19:15.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:19:15.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:19:15.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:19:15.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:19:15.882 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:15.891+0000 7f9f5764e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:15.919 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:15.927+0000 7f9f5764e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:15.920 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:15.931+0000 7f9f5764e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:16.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T00:19:16.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:19:16.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:19:16.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:19:16.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:19:16.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:16.272 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:19:16.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:19:16.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:16.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:16.486 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:16.495+0000 7f9f5764e7c0 -1 Falling back to public interface 2024-03-25T00:19:16.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:17.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:17.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:17.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:19:17.734 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:19:17.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:17.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:18.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:19.124 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:19.135+0000 7f9f5764e7c0 -1 osd.2 0 log_to_monitors true 2024-03-25T00:19:19.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:19.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:19.162 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:19:19.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:19:19.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:19.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:19.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:20.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:20.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:20.793 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:19:20.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:19:20.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:20.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:21.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:22.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:22.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:22.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:19:22.227 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:19:22.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:22.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:22.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:23.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:23.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:23.656 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:19:23.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:19:23.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:23.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:24.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:25.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:25.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:25.218 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:19:25.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:19:25.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:25.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:25.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:26.636 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:26.636 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:26.637 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:19:26.637 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:19:26.637 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:26.637 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:27.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:28.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:28.057 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:19:28.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:28.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:19:28.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:28.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:28.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:29.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:29.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:29.484 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:19:29.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:19:29.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:29.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:29.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:30.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:30.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:30.988 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:19:30.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:19:30.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:30.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:31.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:32.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:32.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:32.814 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:19:32.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:19:32.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:32.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:33.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:34.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:34.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:34.752 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:19:34.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:19:34.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:34.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:35.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:36.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:36.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:36.765 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:19:36.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:19:36.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:36.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:37.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:38.603 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:38.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:38.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:19:38.604 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:19:38.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:38.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:39.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:40.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:40.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:40.594 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:19:40.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:19:40.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:40.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:41.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:42.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:42.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:42.533 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:19:42.533 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:19:42.533 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:42.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:19:43.081 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1040049980,v1:127.0.0.1:6819/1040049980] [v2:127.0.0.1:6820/1040049980,v1:127.0.0.1:6821/1040049980] exists,up 4298834f-324a-49e0-9eb1-e7d2c17bcdfa 2024-03-25T00:19:43.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:19:43.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:19:43.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:19:43.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:19:43.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-03-25T00:19:43.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:19:43.084 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:19:43.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:19:43.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:19:43.085 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:19:43.085 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:19:43.085 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:19:43.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:19:43.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:19:43.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:19:43.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:19:43.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:19:43.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:19:43.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:19:43.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:19:43.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:19:43.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:19:43.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:19:43.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:19:43.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:19:43.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:19:43.087 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:19:43.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd14afa0-d9b1-4387-8d25-16bc7cfcb565 2024-03-25T00:19:43.087 INFO:tasks.workunit.client.0.smithi080.stdout:add osd3 dd14afa0-d9b1-4387-8d25-16bc7cfcb565 2024-03-25T00:19:43.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 dd14afa0-d9b1-4387-8d25-16bc7cfcb565' 2024-03-25T00:19:43.088 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:19:43.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfwwBmJEFiBxAASi3u4kFxETGW26S7R7y0GA== 2024-03-25T00:19:43.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfwwBmJEFiBxAASi3u4kFxETGW26S7R7y0GA=="}' 2024-03-25T00:19:43.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd14afa0-d9b1-4387-8d25-16bc7cfcb565 -i td/test-erasure-eio/3/new.json 2024-03-25T00:19:43.907 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:19:43.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-03-25T00:19:43.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfwwBmJEFiBxAASi3u4kFxETGW26S7R7y0GA== --osd-uuid dd14afa0-d9b1-4387-8d25-16bc7cfcb565 2024-03-25T00:19:43.963 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:43.970+0000 7f46324817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:43.965 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:43.974+0000 7f46324817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:43.966 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:43.974+0000 7f46324817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:43.967 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:43.974+0000 7f46324817c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:19:43.967 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:43.974+0000 7f46324817c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:19:43.967 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:43.974+0000 7f46324817c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:19:44.049 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:44.058+0000 7f46324817c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-03-25T00:19:50.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-03-25T00:19:50.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:19:50.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T00:19:50.172 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd3 key to auth repository 2024-03-25T00:19:50.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:19:50.729 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.3 2024-03-25T00:19:50.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T00:19:50.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:19:50.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:19:50.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:19:50.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:19:50.765 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:50.774+0000 7f22d46ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:50.791 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:50.798+0000 7f22d46ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:50.793 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:50.802+0000 7f22d46ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:19:51.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T00:19:51.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:19:51.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:19:51.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:19:51.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:19:51.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:51.148 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:19:51.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:19:51.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:51.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:19:51.338 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:51.346+0000 7f22d46ca7c0 -1 Falling back to public interface 2024-03-25T00:19:51.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:52.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:52.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:52.572 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:19:52.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:19:52.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:52.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:19:52.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:53.842 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:19:53.850+0000 7f22d46ca7c0 -1 osd.3 0 log_to_monitors true 2024-03-25T00:19:53.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:53.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:53.998 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:19:53.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:19:53.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:53.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:19:54.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:55.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:55.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:55.516 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:19:55.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:19:55.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:55.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:19:56.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:57.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:57.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:57.065 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:19:57.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:19:57.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:57.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:19:57.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:19:58.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:19:58.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:19:58.698 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:19:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:19:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:19:58.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:19:59.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:00.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:00.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:00.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:20:00.128 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:20:00.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:00.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:00.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:01.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:01.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:01.741 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:20:01.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:20:01.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:01.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:02.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:03.168 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:03.168 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:03.168 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:20:03.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:20:03.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:03.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:03.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:04.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:04.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:04.710 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:20:04.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:20:04.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:04.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:05.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:06.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:06.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:06.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:20:06.127 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:20:06.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:06.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:06.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:07.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:07.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:07.674 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:20:07.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:20:07.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:07.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:08.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:09.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:09.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:09.148 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:20:09.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:20:09.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:09.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:10.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:11.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:11.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:11.065 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:20:11.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:20:11.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:11.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:11.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:12.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:12.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:12.989 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:20:12.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:20:12.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:12.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:14.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:15.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:15.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:15.044 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:20:15.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:20:15.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:15.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:15.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:16.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:16.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:16.473 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:20:16.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:20:16.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:16.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:17.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:18.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:18.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:18.593 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:20:18.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:20:18.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:18.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:19.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:20.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:20.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:20.362 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:20:20.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:20:20.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:20.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:20:20.917 INFO:tasks.workunit.client.0.smithi080.stdout:osd.3 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2423350080,v1:127.0.0.1:6827/2423350080] [v2:127.0.0.1:6828/2423350080,v1:127.0.0.1:6829/2423350080] exists,up dd14afa0-d9b1-4387-8d25-16bc7cfcb565 2024-03-25T00:20:20.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:20:20.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-03-25T00:20:20.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-03-25T00:20:20.921 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:20:20.921 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:20:20.921 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:20:20.921 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:20:20.921 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:20:20.921 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:20:20.921 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:20:20.921 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:20:20.921 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:20:20.921 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:20:20.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:20:20.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:20:20.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=38b55693-00f2-4d58-b8e6-a1bcea1371d2 2024-03-25T00:20:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 38b55693-00f2-4d58-b8e6-a1bcea1371d2' 2024-03-25T00:20:20.924 INFO:tasks.workunit.client.0.smithi080.stdout:add osd4 38b55693-00f2-4d58-b8e6-a1bcea1371d2 2024-03-25T00:20:20.924 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:20:20.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEwwBm3+XrOBAAJ18tGe5aiEoLZfX6lzs8Jg== 2024-03-25T00:20:20.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEwwBm3+XrOBAAJ18tGe5aiEoLZfX6lzs8Jg=="}' 2024-03-25T00:20:20.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 38b55693-00f2-4d58-b8e6-a1bcea1371d2 -i td/test-erasure-eio/4/new.json 2024-03-25T00:20:21.546 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:20:21.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-03-25T00:20:21.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEwwBm3+XrOBAAJ18tGe5aiEoLZfX6lzs8Jg== --osd-uuid 38b55693-00f2-4d58-b8e6-a1bcea1371d2 2024-03-25T00:20:21.601 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:21.609+0000 7fef839777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:20:21.606 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:21.609+0000 7fef839777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:20:21.606 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:21.613+0000 7fef839777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:20:21.607 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:21.613+0000 7fef839777c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:20:21.607 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:21.613+0000 7fef839777c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:20:21.607 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:21.613+0000 7fef839777c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:20:21.713 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:21.721+0000 7fef839777c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-03-25T00:20:28.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-03-25T00:20:28.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:20:28.362 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd4 key to auth repository 2024-03-25T00:20:28.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-25T00:20:28.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:20:28.913 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.4 2024-03-25T00:20:28.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-25T00:20:28.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:20:28.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:20:28.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:20:28.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:20:28.949 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:28.953+0000 7f12bf9d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:20:28.967 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:28.973+0000 7f12bf9d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:20:28.968 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:28.973+0000 7f12bf9d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:20:29.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-25T00:20:29.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:20:29.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-25T00:20:29.372 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:20:29.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:20:29.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:20:29.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:29.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:20:29.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:29.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:29.530 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:29.537+0000 7f12bf9d67c0 -1 Falling back to public interface 2024-03-25T00:20:29.801 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:30.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:30.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:30.803 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:20:30.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:20:30.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:30.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:31.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:31.900 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:20:31.905+0000 7f12bf9d67c0 -1 osd.4 0 log_to_monitors true 2024-03-25T00:20:32.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:32.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:32.226 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:20:32.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:20:32.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:32.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:32.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:33.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:33.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:33.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:20:33.732 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:20:33.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:33.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:34.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:35.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:35.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:35.366 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:20:35.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:20:35.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:35.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:35.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:36.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:36.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:36.880 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:20:36.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:20:36.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:36.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:37.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:38.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:38.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:38.354 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:20:38.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:20:38.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:38.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:38.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:39.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:39.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:39.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:20:39.761 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:20:39.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:39.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:40.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:41.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:41.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:41.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:20:41.191 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:20:41.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:41.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:41.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:42.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:42.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:42.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:20:42.659 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:20:42.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:42.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:43.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:44.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:44.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:44.105 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:20:44.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:20:44.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:44.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:44.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:45.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:45.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:45.692 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:20:45.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:20:45.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:45.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:46.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:47.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:47.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:47.292 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:20:47.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:20:47.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:47.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:48.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:49.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:49.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:49.070 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:20:49.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:20:49.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:49.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:49.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:50.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:50.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:50.597 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:20:50.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:20:50.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:50.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:51.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:52.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:52.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:52.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:20:52.646 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:20:52.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:52.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:53.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:54.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:54.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:54.102 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:20:54.102 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:20:54.102 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:54.102 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:55.168 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:56.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:56.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:56.170 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:20:56.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:20:56.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:56.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:56.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:20:57.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:20:57.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:20:57.885 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:20:57.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:20:57.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:20:57.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:20:58.387 INFO:tasks.workunit.client.0.smithi080.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3441704461,v1:127.0.0.1:6835/3441704461] [v2:127.0.0.1:6836/3441704461,v1:127.0.0.1:6837/3441704461] exists,up 38b55693-00f2-4d58-b8e6-a1bcea1371d2 2024-03-25T00:20:58.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:20:58.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:20:58.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:20:58.389 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-03-25T00:20:58.389 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:20:58.389 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:20:58.390 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:20:58.390 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:20:58.390 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:20:58.390 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.0.asok 2024-03-25T00:20:58.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-03-25T00:20:58.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.0.asok log flush 2024-03-25T00:20:58.498 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:20:58.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-03-25T00:20:58.516 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:18:06.528+0000 7f119fad17c0 0 load: jerasure load: lrc 2024-03-25T00:20:58.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:20:58.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2024-03-25T00:20:58.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-03-25T00:20:58.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:20:58.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-03-25T00:20:58.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-03-25T00:20:58.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-03-25T00:20:58.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-03-25T00:20:58.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-03-25T00:20:58.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-03-25T00:20:59.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-03-25T00:20:59.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-03-25T00:21:00.700 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' already exists 2024-03-25T00:21:00.717 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:21:01.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-03-25T00:21:01.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:21:01.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:21:01.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:21:01.720 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:21:01.721 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:21:01.721 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:21:01.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:21:01.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:21:01.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:21:01.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:21:01.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:21:01.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:21:01.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:21:01.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:21:01.844 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:21:02.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:21:02.332 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:21:02.332 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:21:02.332 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:21:02.332 INFO:tasks.workunit.client.0.smithi080.stderr:4' 2024-03-25T00:21:02.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:21:02.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:21:02.332 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:21:02.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803808 2024-03-25T00:21:02.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803808 2024-03-25T00:21:02.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808' 2024-03-25T00:21:02.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:21:02.495 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:21:02.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542174 2024-03-25T00:21:02.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542174 2024-03-25T00:21:02.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-60129542174' 2024-03-25T00:21:02.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:21:02.648 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:21:02.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345946 2024-03-25T00:21:02.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345946 2024-03-25T00:21:02.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-60129542174 2-85899345946' 2024-03-25T00:21:02.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:21:02.805 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:21:02.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=133143986194 2024-03-25T00:21:02.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 133143986194 2024-03-25T00:21:02.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-60129542174 2-85899345946 3-133143986194' 2024-03-25T00:21:02.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:21:02.959 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:21:03.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659143 2024-03-25T00:21:03.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659143 2024-03-25T00:21:03.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-60129542174 2-85899345946 3-133143986194 4-176093659143' 2024-03-25T00:21:03.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:21:03.113 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803808 2024-03-25T00:21:03.113 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:21:03.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:21:03.114 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803808 2024-03-25T00:21:03.114 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:21:03.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803808 2024-03-25T00:21:03.115 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803808 2024-03-25T00:21:03.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-03-25T00:21:03.116 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:21:03.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803806 -lt 25769803808 2024-03-25T00:21:03.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:21:04.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:21:04.501 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:21:04.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803809 -lt 25769803808 2024-03-25T00:21:04.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:21:04.965 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-60129542174 2024-03-25T00:21:04.966 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:21:04.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:21:04.967 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-60129542174 2024-03-25T00:21:04.968 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:21:04.968 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542174 2024-03-25T00:21:04.968 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 60129542174' 2024-03-25T00:21:04.969 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 60129542174 2024-03-25T00:21:04.969 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:21:05.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542175 -lt 60129542174 2024-03-25T00:21:05.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:21:05.381 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345946 2024-03-25T00:21:05.382 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:21:05.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:21:05.384 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345946 2024-03-25T00:21:05.384 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:21:05.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345946 2024-03-25T00:21:05.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345946' 2024-03-25T00:21:05.386 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 85899345946 2024-03-25T00:21:05.386 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:21:05.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345946 -lt 85899345946 2024-03-25T00:21:05.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:21:05.830 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:21:05.830 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-133143986194 2024-03-25T00:21:05.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:21:05.833 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:21:05.833 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-133143986194 2024-03-25T00:21:05.833 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=133143986194 2024-03-25T00:21:05.834 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 133143986194 2024-03-25T00:21:05.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 133143986194' 2024-03-25T00:21:05.834 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:21:06.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 133143986195 -lt 133143986194 2024-03-25T00:21:06.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:21:06.248 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:21:06.248 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659143 2024-03-25T00:21:06.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:21:06.250 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659143 2024-03-25T00:21:06.250 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:21:06.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659143 2024-03-25T00:21:06.251 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 176093659143 2024-03-25T00:21:06.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659143' 2024-03-25T00:21:06.252 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:21:06.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659144 -lt 176093659143 2024-03-25T00:21:06.675 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:21:06.676 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:21:06.676 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:21:07.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:21:07.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:21:07.222 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:21:07.222 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:21:07.222 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:21:07.223 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:21:07.223 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:21:07.223 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:21:07.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:21:07.609 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:21:07.609 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:21:07.609 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:21:08.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:21:08.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:21:08.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:21:08.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:525: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-03-25T00:21:08.563 INFO:tasks.workunit.client.0.smithi080.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T00:21:08.564 INFO:tasks.workunit.client.0.smithi080.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:21:01.781358+0000 0'0 49:15 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-03-25T00:21:00.453631+0000 0'0 2024-03-25T00:21:00.453631+0000 0 0 periodic scrub scheduled @ 2024-03-26T04:23:44.949915+0000 0 0 2024-03-25T00:21:08.565 INFO:tasks.workunit.client.0.smithi080.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:20:59.784864+0000 0'0 49:21 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T07:58:37.272242+0000 0 0 2024-03-25T00:21:08.565 INFO:tasks.workunit.client.0.smithi080.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:20:59.390852+0000 0'0 49:110 [0,4,1] 0 [0,4,1] 0 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T06:37:30.204071+0000 0 0 2024-03-25T00:21:08.565 INFO:tasks.workunit.client.0.smithi080.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:20:59.703329+0000 0'0 49:21 [4,3,0] 4 [4,3,0] 4 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T02:30:00.066648+0000 0 0 2024-03-25T00:21:08.565 INFO:tasks.workunit.client.0.smithi080.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:19:43.004809+0000 0'0 49:75 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T08:12:40.249203+0000 0 0 2024-03-25T00:21:08.565 INFO:tasks.workunit.client.0.smithi080.stdout: 2024-03-25T00:21:08.565 INFO:tasks.workunit.client.0.smithi080.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T00:21:08.565 INFO:tasks.workunit.client.0.smithi080.stderr:dumped pgs 2024-03-25T00:21:08.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:527: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-03-25T00:21:08.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:21:08.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:21:08.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-03-25T00:21:08.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:21:08.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:21:08.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:21:08.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:21:08.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:21:08.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:21:08.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:21:08.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:21:08.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:08.769 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2024-03-25T00:21:08.769 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-03-25T00:21:08.769 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-03-25T00:21:08.770 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:21:08.770 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T00:21:09.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: local primary=3 2024-03-25T00:21:09.171 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2024-03-25T00:21:09.171 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:21:09.171 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-03-25T00:21:09.172 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:21:09.172 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:21:09.600 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:21:09.600 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:21:09.600 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:21:09.600 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:21:09.600 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-25T00:21:09.601 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-03-25T00:21:09.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-03-25T00:21:09.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: local -a initial_osds 2024-03-25T00:21:09.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local last_osd=2 2024-03-25T00:21:09.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-03-25T00:21:09.602 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:21:09.602 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:21:09.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:21:09.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:21:09.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:21:09.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:21:09.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: ceph osd down 2 2024-03-25T00:21:10.677 INFO:tasks.workunit.client.0.smithi080.stderr:osd.2 is already down. 2024-03-25T00:21:10.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:534: TEST_ec_backfill_unfound: ceph osd out 2 2024-03-25T00:21:11.625 INFO:tasks.workunit.client.0.smithi080.stderr:osd.2 is already out. 2024-03-25T00:21:11.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:536: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-03-25T00:21:12.190 INFO:tasks.workunit.client.0.smithi080.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T00:21:12.191 INFO:tasks.workunit.client.0.smithi080.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 peering 2024-03-25T00:21:10.026451+0000 49'1 49:17 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-03-25T00:21:00.453631+0000 0'0 2024-03-25T00:21:00.453631+0000 0 0 no scrub is scheduled 0 0 2024-03-25T00:21:12.191 INFO:tasks.workunit.client.0.smithi080.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:20:59.784864+0000 0'0 49:21 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T07:58:37.272242+0000 0 0 2024-03-25T00:21:12.191 INFO:tasks.workunit.client.0.smithi080.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:20:59.390852+0000 0'0 49:110 [0,4,1] 0 [0,4,1] 0 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T06:37:30.204071+0000 0 0 2024-03-25T00:21:12.191 INFO:tasks.workunit.client.0.smithi080.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:20:59.703329+0000 0'0 49:21 [4,3,0] 4 [4,3,0] 4 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T02:30:00.066648+0000 0 0 2024-03-25T00:21:12.191 INFO:tasks.workunit.client.0.smithi080.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:19:43.004809+0000 0'0 49:76 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T08:12:40.249203+0000 0 0 2024-03-25T00:21:12.191 INFO:tasks.workunit.client.0.smithi080.stdout: 2024-03-25T00:21:12.191 INFO:tasks.workunit.client.0.smithi080.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T00:21:12.191 INFO:tasks.workunit.client.0.smithi080.stderr:dumped pgs 2024-03-25T00:21:12.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-03-25T00:21:12.209 INFO:tasks.workunit.client.0.smithi080.stderr:4+0 records in 2024-03-25T00:21:12.209 INFO:tasks.workunit.client.0.smithi080.stderr:4+0 records out 2024-03-25T00:21:12.209 INFO:tasks.workunit.client.0.smithi080.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000624901 s, 6.6 MB/s 2024-03-25T00:21:12.209 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: seq 1 300 2024-03-25T00:21:12.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:12.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:12.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:12.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:12.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:12.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:13.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:13.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:13.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:13.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:13.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:13.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:13.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:13.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:14.111 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:14.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:14.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:14.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:14.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:14.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:14.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:14.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:15.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:15.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:15.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:15.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:15.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:15.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:15.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:15.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:16.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:16.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:16.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:16.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:16.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:16.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:16.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:16.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:17.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:17.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:17.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:17.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:17.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:17.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:17.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:17.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:18.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:18.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:18.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:18.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:18.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:18.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:18.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:18.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:19.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:19.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:19.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:19.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:19.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:19.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:19.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:19.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:20.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:20.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:20.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:20.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:20.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:20.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:20.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:20.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:21.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:21.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:21.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:21.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:21.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:21.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:21.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:21.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:22.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:22.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:22.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:22.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:22.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:22.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:22.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:22.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:23.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:23.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:23.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:23.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:23.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:23.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:23.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:23.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:24.241 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:24.241 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:24.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:24.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:24.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:24.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:25.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:25.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:25.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:25.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:25.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:25.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:25.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:25.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:26.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:26.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:26.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:26.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:26.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:26.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:26.968 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:26.968 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:27.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:27.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:27.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:27.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:27.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:27.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:28.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:28.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:28.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:28.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:28.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:28.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:28.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:28.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:29.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:29.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:29.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:29.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:29.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:29.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:29.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:29.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:30.064 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:30.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:30.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:30.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:30.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:30.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:30.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:30.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:31.241 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:31.241 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:31.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:31.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:31.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:31.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:31.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:31.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:32.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:32.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:32.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:32.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:32.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:32.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:33.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:33.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:33.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:33.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:33.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:33.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:33.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:33.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:34.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:34.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:34.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:34.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:34.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:34.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:34.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:34.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:35.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:35.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:35.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:35.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:35.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:35.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:35.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:35.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:36.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:36.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:36.603 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:36.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:36.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:36.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:37.074 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:37.074 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:37.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:37.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:37.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:37.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:37.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:37.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:38.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:38.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:38.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:38.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:38.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:38.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:38.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:38.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:39.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:39.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:39.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:39.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:39.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:39.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:39.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:39.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:40.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:40.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:40.315 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:40.315 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:40.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:40.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:40.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:40.836 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:41.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:41.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:41.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:41.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:41.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:41.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:41.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:41.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:42.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:42.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:42.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:42.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:42.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:42.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:42.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:42.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:43.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:43.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:43.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:43.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:43.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:43.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:43.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:43.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:44.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:44.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:44.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:44.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:44.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:44.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:44.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:44.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:45.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:45.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:45.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:45.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:45.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:45.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:45.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:45.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:46.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:46.260 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:46.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:46.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:46.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:46.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:47.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:47.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:47.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:47.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:47.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:47.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:47.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:47.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:48.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:48.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:48.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:48.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:48.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:48.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:48.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:48.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:49.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:49.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:49.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:49.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:49.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:49.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:49.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:49.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:50.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:50.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:50.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:50.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:50.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:50.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:50.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:50.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:51.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:51.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:51.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:51.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:51.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:51.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:51.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:51.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:52.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:52.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:52.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:52.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:52.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:52.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:52.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:52.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:53.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:53.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:53.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:53.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:53.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:53.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:53.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:53.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:54.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:54.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:54.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:54.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:54.627 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:54.627 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:54.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:54.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:55.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:55.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:55.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:55.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:55.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:55.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:55.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:55.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:56.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:56.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:56.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:56.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:56.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:56.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:56.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:56.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:57.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:57.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:57.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:57.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:57.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:57.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:57.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:57.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:58.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:58.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:58.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:58.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:58.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:58.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:58.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:58.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:59.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:59.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:59.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:59.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:59.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:59.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2024-03-25T00:21:59.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:21:59.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:00.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:00.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:00.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:00.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:00.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:00.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:01.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:01.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:01.278 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:01.278 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:01.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:01.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:01.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:01.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:02.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:02.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:02.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:02.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:02.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:02.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:03.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:03.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:03.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:03.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:03.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:03.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:03.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:03.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:04.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:04.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:04.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:04.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:04.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:04.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:04.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:04.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:05.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:05.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:05.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:05.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:05.539 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:05.539 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:05.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:05.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:06.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:06.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:06.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:06.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:06.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:06.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:06.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:06.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:07.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:07.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:07.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:07.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:07.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:07.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:07.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:07.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:08.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:08.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:08.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:08.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:08.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:08.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:08.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:08.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:09.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:09.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:09.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:09.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:09.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:09.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:10.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:10.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:10.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:10.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:10.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:10.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:10.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:10.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:11.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:11.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:11.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:11.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:11.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:11.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:11.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:11.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:12.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:12.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:12.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:12.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:12.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:12.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:12.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:12.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:13.094 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:13.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:13.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:13.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:13.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:13.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:13.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:13.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:14.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:14.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:14.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:14.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:14.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:14.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:14.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:14.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:15.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:15.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:15.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:15.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:15.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:15.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:15.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:15.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:16.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:16.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:16.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:16.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:16.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:16.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:16.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:16.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:17.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:17.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:17.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:17.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:17.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:17.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:18.007 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:18.007 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:18.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:18.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:18.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:18.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:18.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:18.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:19.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:19.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:19.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:19.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:19.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:19.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:19.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:19.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:20.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:20.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:20.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:20.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:20.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:20.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:20.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:20.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:21.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:21.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:21.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:21.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:21.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:21.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:21.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:21.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:22.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:22.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:22.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:22.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:22.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:22.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:22.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:22.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:23.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:23.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:23.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:23.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:23.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:23.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:23.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:23.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:24.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:24.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:24.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:24.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:24.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:24.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:24.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:24.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:25.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:25.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:25.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:25.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:25.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:25.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:25.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:25.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:26.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:26.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:26.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:26.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:26.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:26.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:26.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:26.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:27.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:27.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:27.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:27.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:27.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:27.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:27.868 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:27.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:28.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:28.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:28.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:28.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:28.641 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:28.641 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:28.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:28.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:29.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:29.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:29.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:22:29.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2024-03-25T00:22:29.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:544: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2024-03-25T00:22:29.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-03-25T00:22:29.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-25T00:22:29.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-25T00:22:29.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-25T00:22:29.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-03-25T00:22:29.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-25T00:22:29.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj250 2024-03-25T00:22:29.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-25T00:22:29.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-03-25T00:22:29.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-25T00:22:29.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-25T00:22:29.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-25T00:22:29.559 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj250 2024-03-25T00:22:29.559 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:22:29.559 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-03-25T00:22:29.559 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-03-25T00:22:29.559 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:22:29.974 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:22:29.974 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:22:29.974 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:22:29.974 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:22:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:2147483647' 2024-03-25T00:22:29.975 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-03-25T00:22:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-03-25T00:22:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-25T00:22:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-03-25T00:22:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-03-25T00:22:29.975 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-03-25T00:22:29.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-25T00:22:29.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-03-25T00:22:29.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T00:22:29.977 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-03-25T00:22:29.977 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-25T00:22:29.977 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-25T00:22:29.977 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T00:22:29.977 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-03-25T00:22:29.977 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-03-25T00:22:29.977 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-03-25T00:22:29.977 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:22:29.977 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:22:29.977 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:22:29.978 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.3.asok 2024-03-25T00:22:29.978 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81310/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-03-25T00:22:30.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T00:22:30.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-25T00:22:30.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-25T00:22:30.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-03-25T00:22:30.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-03-25T00:22:30.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-03-25T00:22:30.086 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:22:30.086 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:22:30.086 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:22:30.086 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.3.asok 2024-03-25T00:22:30.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-25T00:22:30.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj250 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-25T00:22:30.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-25T00:22:30.214 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj250 2024-03-25T00:22:30.214 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:22:30.214 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-03-25T00:22:30.214 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-03-25T00:22:30.214 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:22:30.670 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:22:30.670 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:22:30.670 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:22:30.670 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:22:30.670 INFO:tasks.workunit.client.0.smithi080.stderr:2147483647' 2024-03-25T00:22:30.670 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-03-25T00:22:30.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-03-25T00:22:30.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-25T00:22:30.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-25T00:22:30.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-03-25T00:22:30.671 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/1/type 2024-03-25T00:22:30.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-25T00:22:30.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-25T00:22:30.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T00:22:30.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-25T00:22:30.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-25T00:22:30.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-25T00:22:30.672 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-25T00:22:30.672 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T00:22:30.672 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-25T00:22:30.672 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-25T00:22:30.673 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:22:30.673 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:22:30.673 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:22:30.673 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.1.asok 2024-03-25T00:22:30.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81310/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-25T00:22:30.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T00:22:30.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-25T00:22:30.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-25T00:22:30.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-25T00:22:30.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-25T00:22:30.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-25T00:22:30.784 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:22:30.784 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:22:30.785 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:22:30.785 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.1.asok 2024-03-25T00:22:30.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-25T00:22:30.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2024-03-25T00:22:30.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-03-25T00:22:30.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:22:30.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:22:30.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:22:30.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:22:30.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:22:30.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:22:30.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:22:30.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:22:30.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:22:30.912 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:22:30.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-25T00:22:30.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:22:30.913 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-03-25T00:22:30.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-25T00:22:30.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:22:30.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:22:30.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:22:30.941 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:22:30.946+0000 7fde3db257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:22:30.944 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:22:30.950+0000 7fde3db257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:22:30.945 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:22:30.950+0000 7fde3db257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:22:31.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-25T00:22:31.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:22:31.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:22:31.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:22:31.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:22:31.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:22:31.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:22:31.328 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:22:31.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:22:31.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:22:31.492 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:22:31.498+0000 7fde3db257c0 -1 Falling back to public interface 2024-03-25T00:22:31.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:22:32.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:22:32.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:22:32.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:22:32.741 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:22:32.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:22:32.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:22:33.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:22:34.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:22:34.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:22:34.216 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:22:34.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:22:34.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:22:34.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:22:34.406 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:22:34.410+0000 7fde3db257c0 -1 osd.2 49 log_to_monitors true 2024-03-25T00:22:34.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:22:35.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:22:35.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:22:35.645 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:22:35.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:22:35.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:22:35.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:22:36.245 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up out weight 0 up_from 57 up_thru 22 down_at 50 last_clean_interval [20,49) [v2:127.0.0.1:6818/1193960520,v1:127.0.0.1:6819/1193960520] [v2:127.0.0.1:6820/1193960520,v1:127.0.0.1:6821/1193960520] exists,up 4298834f-324a-49e0-9eb1-e7d2c17bcdfa 2024-03-25T00:22:36.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:22:36.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:22:36.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:22:36.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: ceph osd in 2 2024-03-25T00:22:37.373 INFO:tasks.workunit.client.0.smithi080.stderr:osd.2 is already in. 2024-03-25T00:22:37.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: sleep 15 2024-03-25T00:22:52.396 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: seq 1 240 2024-03-25T00:22:52.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-03-25T00:22:52.398 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-03-25T00:22:52.398 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-03-25T00:22:52.398 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-03-25T00:22:52.399 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-03-25T00:22:52.399 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-03-25T00:22:52.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:52.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:52.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-03-25T00:22:52.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-03-25T00:22:52.818 INFO:tasks.workunit.client.0.smithi080.stdout:active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:52.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:52.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-03-25T00:22:53.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-03-25T00:22:53.820 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-03-25T00:22:53.820 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-03-25T00:22:53.821 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-03-25T00:22:53.821 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-03-25T00:22:53.821 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-03-25T00:22:54.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:54.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:54.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-03-25T00:22:54.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-03-25T00:22:54.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:54.235 INFO:tasks.workunit.client.0.smithi080.stdout:active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:54.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-03-25T00:22:55.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-03-25T00:22:55.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-03-25T00:22:55.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-03-25T00:22:55.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-03-25T00:22:55.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-03-25T00:22:55.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-03-25T00:22:55.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:55.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:55.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-03-25T00:22:55.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-03-25T00:22:55.663 INFO:tasks.workunit.client.0.smithi080.stdout:active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:55.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:55.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-03-25T00:22:56.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-03-25T00:22:56.665 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-03-25T00:22:56.665 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-03-25T00:22:56.665 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-03-25T00:22:56.665 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-03-25T00:22:56.665 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-03-25T00:22:57.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:57.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:57.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-03-25T00:22:57.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-03-25T00:22:57.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:57.124 INFO:tasks.workunit.client.0.smithi080.stdout:active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:57.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-03-25T00:22:58.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-03-25T00:22:58.126 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-03-25T00:22:58.126 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-03-25T00:22:58.126 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-03-25T00:22:58.127 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-03-25T00:22:58.127 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-03-25T00:22:58.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:58.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:58.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-03-25T00:22:58.606 INFO:tasks.workunit.client.0.smithi080.stdout:active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:58.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-03-25T00:22:58.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:22:58.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-03-25T00:22:59.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-03-25T00:22:59.609 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-03-25T00:22:59.609 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-03-25T00:22:59.609 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-03-25T00:22:59.609 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-03-25T00:22:59.610 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-03-25T00:23:00.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-03-25T00:23:00.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:23:00.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-03-25T00:23:00.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-03-25T00:23:00.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:23:00.010 INFO:tasks.workunit.client.0.smithi080.stdout:active+undersized+degraded+remapped+backfilling 2024-03-25T00:23:00.011 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-03-25T00:23:01.011 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-03-25T00:23:01.013 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-03-25T00:23:01.013 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-03-25T00:23:01.013 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-03-25T00:23:01.013 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-03-25T00:23:01.013 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-03-25T00:23:01.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-03-25T00:23:01.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:23:01.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-03-25T00:23:01.418 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-03-25T00:23:01.419 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-03-25T00:23:01.419 INFO:tasks.workunit.client.0.smithi080.stdout:active+undersized+degraded+remapped+backfilling 2024-03-25T00:23:01.419 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-03-25T00:23:02.420 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-03-25T00:23:02.421 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-03-25T00:23:02.421 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-03-25T00:23:02.421 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-03-25T00:23:02.422 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-03-25T00:23:02.422 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-03-25T00:23:02.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-03-25T00:23:02.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-03-25T00:23:02.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-03-25T00:23:02.828 INFO:tasks.workunit.client.0.smithi080.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-03-25T00:23:02.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2024-03-25T00:23:02.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: break 2024-03-25T00:23:02.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:562: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-03-25T00:23:03.196 INFO:tasks.workunit.client.0.smithi080.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T00:23:03.197 INFO:tasks.workunit.client.0.smithi080.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-03-25T00:22:59.564020+0000 55'301 64:900 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2024-03-25T00:21:00.453631+0000 0'0 2024-03-25T00:21:00.453631+0000 0 0 no scrub is scheduled 0 0 2024-03-25T00:23:03.197 INFO:tasks.workunit.client.0.smithi080.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:22:38.177577+0000 0'0 64:64 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T11:30:55.780744+0000 0 0 2024-03-25T00:23:03.197 INFO:tasks.workunit.client.0.smithi080.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:20:59.390852+0000 0'0 64:139 [0,4,1] 0 [0,4,1] 0 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T06:37:30.204071+0000 0 0 2024-03-25T00:23:03.197 INFO:tasks.workunit.client.0.smithi080.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:20:59.703329+0000 0'0 64:50 [4,3,0] 4 [4,3,0] 4 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T02:30:00.066648+0000 0 0 2024-03-25T00:23:03.197 INFO:tasks.workunit.client.0.smithi080.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:22:38.177604+0000 0'0 64:114 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T00:32:00.120375+0000 0 0 2024-03-25T00:23:03.197 INFO:tasks.workunit.client.0.smithi080.stdout: 2024-03-25T00:23:03.197 INFO:tasks.workunit.client.0.smithi080.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T00:23:03.197 INFO:tasks.workunit.client.0.smithi080.stderr:dumped pgs 2024-03-25T00:23:03.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:563: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-03-25T00:23:03.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:23:03.213 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:23:03.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:23:03.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:23:03.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:23:04.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:23:04.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: sleep 5 2024-03-25T00:23:09.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:566: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-03-25T00:23:09.926 INFO:tasks.workunit.client.0.smithi080.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T00:23:09.926 INFO:tasks.workunit.client.0.smithi080.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2024-03-25T00:23:04.896887+0000 55'301 67:914 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-03-25T00:21:00.453631+0000 0'0 2024-03-25T00:21:00.453631+0000 0 0 no scrub is scheduled 0 0 2024-03-25T00:23:09.926 INFO:tasks.workunit.client.0.smithi080.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:22:38.177577+0000 0'0 64:64 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T11:30:55.780744+0000 0 0 2024-03-25T00:23:09.926 INFO:tasks.workunit.client.0.smithi080.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:20:59.390852+0000 0'0 67:145 [0,4,1] 0 [0,4,1] 0 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T06:37:30.204071+0000 0 0 2024-03-25T00:23:09.926 INFO:tasks.workunit.client.0.smithi080.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:20:59.703329+0000 0'0 64:50 [4,3,0] 4 [4,3,0] 4 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T02:30:00.066648+0000 0 0 2024-03-25T00:23:09.926 INFO:tasks.workunit.client.0.smithi080.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:22:38.177604+0000 0'0 64:114 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T00:17:57.738040+0000 0'0 2024-03-25T00:17:57.738040+0000 0 0 periodic scrub scheduled @ 2024-03-26T00:32:00.120375+0000 0 0 2024-03-25T00:23:09.926 INFO:tasks.workunit.client.0.smithi080.stdout: 2024-03-25T00:23:09.926 INFO:tasks.workunit.client.0.smithi080.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T00:23:09.927 INFO:tasks.workunit.client.0.smithi080.stderr:dumped pgs 2024-03-25T00:23:09.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-03-25T00:23:10.092 INFO:tasks.workunit.client.0.smithi080.stdout:{ 2024-03-25T00:23:10.092 INFO:tasks.workunit.client.0.smithi080.stdout: "num_missing": 1, 2024-03-25T00:23:10.092 INFO:tasks.workunit.client.0.smithi080.stdout: "num_unfound": 1, 2024-03-25T00:23:10.092 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": [ 2024-03-25T00:23:10.092 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "oid": { 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "oid": "obj250", 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "key": "", 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "snapid": -2, 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "hash": 2249616407, 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "max": 0, 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "pool": 2, 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "namespace": "" 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "need": "55'251", 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "have": "0'0", 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "flags": "none", 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-03-25T00:23:10.093 INFO:tasks.workunit.client.0.smithi080.stdout: "locations": [ 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: ] 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "NotRecovering", 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: "available_might_have_unfound": true, 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: "might_have_unfound": [ 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: "osd": "2(4)", 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: "status": "osd is down" 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout: "more": false 2024-03-25T00:23:10.094 INFO:tasks.workunit.client.0.smithi080.stdout:} 2024-03-25T00:23:10.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph pg 2.0 query 2024-03-25T00:23:10.260 INFO:tasks.workunit.client.0.smithi080.stdout:{ 2024-03-25T00:23:10.260 INFO:tasks.workunit.client.0.smithi080.stdout: "snap_trimq": "[]", 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: "snap_trimq_len": 0, 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch": 67, 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: "up": [ 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: 2147483647 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: "acting": [ 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:23:10.261 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: 2147483647 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: "acting_recovery_backfill": [ 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: "1(1)", 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: "3(0)", 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: "info": { 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: "pgid": "2.0s0", 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: "shared": "0", 2024-03-25T00:23:10.262 INFO:tasks.workunit.client.0.smithi080.stdout: "last_update": "55'301", 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "last_complete": "55'250", 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "log_tail": "55'200", 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "last_user_version": 301, 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "last_backfill": "MAX", 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [], 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "history": { 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_created": 47, 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_pool_created": 47, 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 67, 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_started": 65, 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 48, 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_clean": 47, 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_split": 0, 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_marked_full": 0, 2024-03-25T00:23:10.263 INFO:tasks.workunit.client.0.smithi080.stdout: "same_up_since": 65, 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "same_interval_since": 65, 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "same_primary_since": 47, 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "prior_readable_until_ub": 0 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "stats": { 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "version": "55'301", 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_seq": 915, 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_epoch": 67, 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-03-25T00:23:10.264 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fresh": "2024-03-25T00:23:05.368416+0000", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "last_change": "2024-03-25T00:23:04.896887+0000", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "last_active": "2024-03-25T00:23:05.368416+0000", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "last_peered": "2024-03-25T00:23:05.368416+0000", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean": "2024-03-25T00:21:08.775443+0000", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_active": "2024-03-25T00:23:04.869624+0000", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_peered": "2024-03-25T00:23:04.869624+0000", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "last_unstale": "2024-03-25T00:23:05.368416+0000", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "last_undegraded": "2024-03-25T00:23:04.562463+0000", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fullsized": "2024-03-25T00:23:04.561725+0000", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "mapping_epoch": 65, 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "log_start": "55'200", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_start": "55'200", 2024-03-25T00:23:10.265 INFO:tasks.workunit.client.0.smithi080.stdout: "created": 47, 2024-03-25T00:23:10.266 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 48, 2024-03-25T00:23:10.266 INFO:tasks.workunit.client.0.smithi080.stdout: "parent": "0.0", 2024-03-25T00:23:10.266 INFO:tasks.workunit.client.0.smithi080.stdout: "parent_split_bits": 0, 2024-03-25T00:23:10.266 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:23:10.266 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_scrubbed": 0, 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "log_size": 101, 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "log_dups_size": 200, 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_size": 101, 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "stats_invalid": false, 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "dirty_stats_invalid": false, 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "omap_stats_invalid": false, 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_stats_invalid": false, 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_bytes_stats_invalid": false, 2024-03-25T00:23:10.267 INFO:tasks.workunit.client.0.smithi080.stdout: "pin_stats_invalid": false, 2024-03-25T00:23:10.268 INFO:tasks.workunit.client.0.smithi080.stdout: "manifest_stats_invalid": false, 2024-03-25T00:23:10.268 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrimq_len": 0, 2024-03-25T00:23:10.268 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_duration": 0, 2024-03-25T00:23:10.269 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_schedule": "no scrub is scheduled", 2024-03-25T00:23:10.269 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_duration": 0, 2024-03-25T00:23:10.269 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_trimmed": 0, 2024-03-25T00:23:10.269 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrim_duration": 0, 2024-03-25T00:23:10.269 INFO:tasks.workunit.client.0.smithi080.stdout: "stat_sum": { 2024-03-25T00:23:10.269 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes": 1232896, 2024-03-25T00:23:10.270 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects": 301, 2024-03-25T00:23:10.270 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_clones": 0, 2024-03-25T00:23:10.270 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_copies": 1505, 2024-03-25T00:23:10.270 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing_on_primary": 1, 2024-03-25T00:23:10.271 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing": 1, 2024-03-25T00:23:10.271 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_degraded": 303, 2024-03-25T00:23:10.271 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_misplaced": 0, 2024-03-25T00:23:10.271 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_unfound": 1, 2024-03-25T00:23:10.271 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_dirty": 301, 2024-03-25T00:23:10.272 INFO:tasks.workunit.client.0.smithi080.stdout: "num_whiteouts": 0, 2024-03-25T00:23:10.272 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read": 0, 2024-03-25T00:23:10.272 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read_kb": 0, 2024-03-25T00:23:10.272 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write": 301, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write_kb": 1204, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_scrub_errors": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_shallow_scrub_errors": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_deep_scrub_errors": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_recovered": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_recovered": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_keys_recovered": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_omap": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_hit_set_archive": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_hit_set_archive": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_kb": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict": 0, 2024-03-25T00:23:10.273 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_kb": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_promote": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_high": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_low": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_some": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_full": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_pinned": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_legacy_snapsets": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_large_omap_objects": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_manifest": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_bytes": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_keys": 0, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_repaired": 0 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.274 INFO:tasks.workunit.client.0.smithi080.stdout: "up": [ 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: 2147483647 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: "acting": [ 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: 2147483647 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.275 INFO:tasks.workunit.client.0.smithi080.stdout: "avail_no_missing": [ 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: "object_location_counts": [ 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": 300 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: "shards": "0(3),4(2)", 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": 1 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.276 INFO:tasks.workunit.client.0.smithi080.stdout: "blocked_by": [], 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: "up_primary": 3, 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: "acting_primary": 3, 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [] 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: "empty": 0, 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: "dne": 0, 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: "incomplete": 0, 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 67, 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: "hit_set_history": { 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: "current_last_update": "0'0", 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: "history": [] 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: "peer_info": [ 2024-03-25T00:23:10.277 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.278 INFO:tasks.workunit.client.0.smithi080.stdout: "peer": "0(3)", 2024-03-25T00:23:10.278 INFO:tasks.workunit.client.0.smithi080.stdout: "pgid": "2.0s3", 2024-03-25T00:23:10.278 INFO:tasks.workunit.client.0.smithi080.stdout: "shared": "3", 2024-03-25T00:23:10.278 INFO:tasks.workunit.client.0.smithi080.stdout: "last_update": "55'301", 2024-03-25T00:23:10.278 INFO:tasks.workunit.client.0.smithi080.stdout: "last_complete": "55'301", 2024-03-25T00:23:10.279 INFO:tasks.workunit.client.0.smithi080.stdout: "log_tail": "55'200", 2024-03-25T00:23:10.279 INFO:tasks.workunit.client.0.smithi080.stdout: "last_user_version": 301, 2024-03-25T00:23:10.279 INFO:tasks.workunit.client.0.smithi080.stdout: "last_backfill": "MAX", 2024-03-25T00:23:10.279 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [], 2024-03-25T00:23:10.279 INFO:tasks.workunit.client.0.smithi080.stdout: "history": { 2024-03-25T00:23:10.279 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_created": 47, 2024-03-25T00:23:10.279 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_pool_created": 47, 2024-03-25T00:23:10.279 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 67, 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_started": 65, 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 48, 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_clean": 47, 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_split": 0, 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_marked_full": 0, 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "same_up_since": 65, 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "same_interval_since": 65, 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "same_primary_since": 47, 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: "prior_readable_until_ub": 0 2024-03-25T00:23:10.280 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "stats": { 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "version": "55'301", 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_seq": 903, 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_epoch": 65, 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "active+undersized+degraded", 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fresh": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "last_change": "2024-03-25T00:21:10.995858+0000", 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "last_active": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "last_peered": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean": "2024-03-25T00:21:08.775443+0000", 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_active": "2024-03-25T00:21:10.995858+0000", 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_peered": "2024-03-25T00:21:10.995858+0000", 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "last_unstale": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "last_undegraded": "2024-03-25T00:21:10.667813+0000", 2024-03-25T00:23:10.281 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fullsized": "2024-03-25T00:21:10.667535+0000", 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "mapping_epoch": 65, 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "log_start": "55'200", 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_start": "55'200", 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "created": 47, 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 48, 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "parent": "0.0", 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "parent_split_bits": 0, 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.282 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_scrubbed": 0, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "log_size": 101, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "log_dups_size": 200, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_size": 101, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "stats_invalid": false, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "dirty_stats_invalid": false, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "omap_stats_invalid": false, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_stats_invalid": false, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_bytes_stats_invalid": false, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "pin_stats_invalid": false, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "manifest_stats_invalid": false, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrimq_len": 0, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_duration": 0, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-03-26T11:51:03.906223+0000", 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_duration": 0, 2024-03-25T00:23:10.283 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_trimmed": 0, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrim_duration": 0, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "stat_sum": { 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes": 1232896, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects": 301, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_clones": 0, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_copies": 1505, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing_on_primary": 0, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing": 0, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_degraded": 301, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_misplaced": 0, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_unfound": 0, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_dirty": 301, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_whiteouts": 0, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read": 0, 2024-03-25T00:23:10.284 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read_kb": 0, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write": 301, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write_kb": 1204, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_scrub_errors": 0, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_shallow_scrub_errors": 0, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_deep_scrub_errors": 0, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_recovered": 0, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_recovered": 0, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_keys_recovered": 0, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_omap": 0, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_hit_set_archive": 0, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_hit_set_archive": 0, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush": 0, 2024-03-25T00:23:10.285 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_kb": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_kb": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_promote": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_high": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_low": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_some": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_full": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_pinned": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_legacy_snapsets": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_large_omap_objects": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_manifest": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_bytes": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_keys": 0, 2024-03-25T00:23:10.286 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_repaired": 0 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: "up": [ 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: 2147483647 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: "acting": [ 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: 2147483647 2024-03-25T00:23:10.287 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: "avail_no_missing": [ 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: "3(0)", 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: "1(1)", 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: "object_location_counts": [ 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": 301 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: "blocked_by": [], 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: "up_primary": 3, 2024-03-25T00:23:10.288 INFO:tasks.workunit.client.0.smithi080.stdout: "acting_primary": 3, 2024-03-25T00:23:10.289 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [] 2024-03-25T00:23:10.289 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.289 INFO:tasks.workunit.client.0.smithi080.stdout: "empty": 0, 2024-03-25T00:23:10.289 INFO:tasks.workunit.client.0.smithi080.stdout: "dne": 0, 2024-03-25T00:23:10.289 INFO:tasks.workunit.client.0.smithi080.stdout: "incomplete": 0, 2024-03-25T00:23:10.289 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 67, 2024-03-25T00:23:10.289 INFO:tasks.workunit.client.0.smithi080.stdout: "hit_set_history": { 2024-03-25T00:23:10.289 INFO:tasks.workunit.client.0.smithi080.stdout: "current_last_update": "0'0", 2024-03-25T00:23:10.289 INFO:tasks.workunit.client.0.smithi080.stdout: "history": [] 2024-03-25T00:23:10.289 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "peer": "1(1)", 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "pgid": "2.0s1", 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "shared": "1", 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "last_update": "55'301", 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "last_complete": "55'301", 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "log_tail": "55'200", 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "last_user_version": 301, 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "last_backfill": "MAX", 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [], 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "history": { 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_created": 47, 2024-03-25T00:23:10.290 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_pool_created": 47, 2024-03-25T00:23:10.291 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 67, 2024-03-25T00:23:10.291 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_started": 65, 2024-03-25T00:23:10.291 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 48, 2024-03-25T00:23:10.291 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_clean": 47, 2024-03-25T00:23:10.291 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_split": 0, 2024-03-25T00:23:10.291 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_marked_full": 0, 2024-03-25T00:23:10.291 INFO:tasks.workunit.client.0.smithi080.stdout: "same_up_since": 65, 2024-03-25T00:23:10.291 INFO:tasks.workunit.client.0.smithi080.stdout: "same_interval_since": 65, 2024-03-25T00:23:10.291 INFO:tasks.workunit.client.0.smithi080.stdout: "same_primary_since": 47, 2024-03-25T00:23:10.291 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "prior_readable_until_ub": 0 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "stats": { 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "version": "55'301", 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_seq": 331, 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_epoch": 55, 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "active+undersized+degraded", 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fresh": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "last_change": "2024-03-25T00:21:10.995858+0000", 2024-03-25T00:23:10.292 INFO:tasks.workunit.client.0.smithi080.stdout: "last_active": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "last_peered": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean": "2024-03-25T00:21:08.775443+0000", 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_active": "2024-03-25T00:21:10.995858+0000", 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_peered": "2024-03-25T00:21:10.995858+0000", 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "last_unstale": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "last_undegraded": "2024-03-25T00:21:10.667813+0000", 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fullsized": "2024-03-25T00:21:10.667535+0000", 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "mapping_epoch": 65, 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "log_start": "55'200", 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_start": "55'200", 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "created": 47, 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 48, 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "parent": "0.0", 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "parent_split_bits": 0, 2024-03-25T00:23:10.293 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_scrubbed": 0, 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "log_size": 101, 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "log_dups_size": 200, 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_size": 101, 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "stats_invalid": false, 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "dirty_stats_invalid": false, 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "omap_stats_invalid": false, 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_stats_invalid": false, 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_bytes_stats_invalid": false, 2024-03-25T00:23:10.294 INFO:tasks.workunit.client.0.smithi080.stdout: "pin_stats_invalid": false, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "manifest_stats_invalid": false, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrimq_len": 0, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_duration": 0, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-03-26T11:51:03.906223+0000", 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_duration": 0, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_trimmed": 0, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrim_duration": 0, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "stat_sum": { 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes": 1232896, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects": 301, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_clones": 0, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_copies": 1505, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing_on_primary": 0, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing": 1, 2024-03-25T00:23:10.295 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_degraded": 301, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_misplaced": 0, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_unfound": 0, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_dirty": 301, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_whiteouts": 0, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read": 0, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read_kb": 0, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write": 301, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write_kb": 1204, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_scrub_errors": 0, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_shallow_scrub_errors": 0, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_deep_scrub_errors": 0, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_recovered": 0, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_recovered": 0, 2024-03-25T00:23:10.296 INFO:tasks.workunit.client.0.smithi080.stdout: "num_keys_recovered": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_omap": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_hit_set_archive": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_hit_set_archive": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_kb": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_kb": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_promote": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_high": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_low": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_some": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_full": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_pinned": 0, 2024-03-25T00:23:10.297 INFO:tasks.workunit.client.0.smithi080.stdout: "num_legacy_snapsets": 0, 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: "num_large_omap_objects": 0, 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_manifest": 0, 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_bytes": 0, 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_keys": 0, 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_repaired": 0 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: "up": [ 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: 2147483647 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: "acting": [ 2024-03-25T00:23:10.298 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: 2147483647 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: "avail_no_missing": [ 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: "3(0)", 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: "1(1)", 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: "object_location_counts": [ 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.299 INFO:tasks.workunit.client.0.smithi080.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": 301 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: "blocked_by": [], 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: "up_primary": 3, 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: "acting_primary": 3, 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [] 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: "empty": 0, 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: "dne": 0, 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: "incomplete": 0, 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 67, 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: "hit_set_history": { 2024-03-25T00:23:10.300 INFO:tasks.workunit.client.0.smithi080.stdout: "current_last_update": "0'0", 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "history": [] 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "peer": "4(2)", 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "pgid": "2.0s2", 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "shared": "2", 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "last_update": "55'301", 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "last_complete": "55'301", 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "log_tail": "55'200", 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "last_user_version": 301, 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "last_backfill": "MAX", 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [], 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "history": { 2024-03-25T00:23:10.301 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_created": 47, 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_pool_created": 47, 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 67, 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_started": 65, 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 48, 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_clean": 47, 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_split": 0, 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_marked_full": 0, 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "same_up_since": 65, 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "same_interval_since": 65, 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "same_primary_since": 47, 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:23:10.302 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "prior_readable_until_ub": 0 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "stats": { 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "version": "55'301", 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_seq": 331, 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_epoch": 55, 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "active+undersized+degraded", 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fresh": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "last_change": "2024-03-25T00:21:10.995858+0000", 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "last_active": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "last_peered": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean": "2024-03-25T00:21:08.775443+0000", 2024-03-25T00:23:10.303 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_active": "2024-03-25T00:21:10.995858+0000", 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_peered": "2024-03-25T00:21:10.995858+0000", 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "last_unstale": "2024-03-25T00:22:29.562919+0000", 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "last_undegraded": "2024-03-25T00:21:10.667813+0000", 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fullsized": "2024-03-25T00:21:10.667535+0000", 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "mapping_epoch": 65, 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "log_start": "55'200", 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_start": "55'200", 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "created": 47, 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 48, 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "parent": "0.0", 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "parent_split_bits": 0, 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:23:10.304 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:21:00.453631+0000", 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_scrubbed": 0, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "log_size": 101, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "log_dups_size": 200, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_size": 101, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "stats_invalid": false, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "dirty_stats_invalid": false, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "omap_stats_invalid": false, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_stats_invalid": false, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_bytes_stats_invalid": false, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "pin_stats_invalid": false, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "manifest_stats_invalid": false, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrimq_len": 0, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_duration": 0, 2024-03-25T00:23:10.305 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-03-26T11:51:03.906223+0000", 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_duration": 0, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_trimmed": 0, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrim_duration": 0, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "stat_sum": { 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes": 1232896, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects": 301, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_clones": 0, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_copies": 1505, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing_on_primary": 0, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing": 0, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_degraded": 301, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_misplaced": 0, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_unfound": 0, 2024-03-25T00:23:10.306 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_dirty": 301, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_whiteouts": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read_kb": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write": 301, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write_kb": 1204, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_scrub_errors": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_shallow_scrub_errors": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_deep_scrub_errors": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_recovered": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_recovered": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_keys_recovered": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_omap": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_hit_set_archive": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_hit_set_archive": 0, 2024-03-25T00:23:10.307 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_kb": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_kb": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_promote": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_high": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_low": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_some": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_full": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_pinned": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_legacy_snapsets": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_large_omap_objects": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_manifest": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_bytes": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_keys": 0, 2024-03-25T00:23:10.308 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_repaired": 0 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: "up": [ 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: 2147483647 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: "acting": [ 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:23:10.309 INFO:tasks.workunit.client.0.smithi080.stdout: 2147483647 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: "avail_no_missing": [ 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: "3(0)", 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: "1(1)", 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: "object_location_counts": [ 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": 301 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.310 INFO:tasks.workunit.client.0.smithi080.stdout: "blocked_by": [], 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: "up_primary": 3, 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: "acting_primary": 3, 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [] 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: "empty": 0, 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: "dne": 0, 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: "incomplete": 0, 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 67, 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: "hit_set_history": { 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: "current_last_update": "0'0", 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: "history": [] 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.311 INFO:tasks.workunit.client.0.smithi080.stdout: "recovery_state": [ 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: "name": "Started/Primary/Active", 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: "enter_time": "2024-03-25T00:23:04.561837+0000", 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: "might_have_unfound": [ 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: "osd": "0(3)", 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: "status": "already probed" 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: "osd": "1(1)", 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: "status": "already probed" 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: "osd": "2(4)", 2024-03-25T00:23:10.312 INFO:tasks.workunit.client.0.smithi080.stdout: "status": "osd is down" 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: "osd": "4(2)", 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: "status": "already probed" 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: "recovery_progress": { 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: "backfill_targets": [], 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: "waiting_on_backfill": [], 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: "last_backfill_started": "MIN", 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: "backfill_info": { 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: "begin": "MIN", 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: "end": "MIN", 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": [] 2024-03-25T00:23:10.313 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: "peer_backfill_info": [], 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: "backfills_in_flight": [], 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: "recovering": [], 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: "pg_backend": { 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: "recovery_ops": [], 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: "read_ops": [] 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: "name": "Started", 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: "enter_time": "2024-03-25T00:23:03.765738+0000" 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:23:10.314 INFO:tasks.workunit.client.0.smithi080.stdout: "scrubber": { 2024-03-25T00:23:10.315 INFO:tasks.workunit.client.0.smithi080.stdout: "active": false, 2024-03-25T00:23:10.315 INFO:tasks.workunit.client.0.smithi080.stdout: "must_scrub": false, 2024-03-25T00:23:10.315 INFO:tasks.workunit.client.0.smithi080.stdout: "must_deep_scrub": false, 2024-03-25T00:23:10.315 INFO:tasks.workunit.client.0.smithi080.stdout: "must_repair": false, 2024-03-25T00:23:10.315 INFO:tasks.workunit.client.0.smithi080.stdout: "need_auto": false, 2024-03-25T00:23:10.315 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_reg_stamp": "2024-03-26T11:51:03.906223+0000", 2024-03-25T00:23:10.315 INFO:tasks.workunit.client.0.smithi080.stdout: "schedule": "no scrub is scheduled" 2024-03-25T00:23:10.315 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:23:10.315 INFO:tasks.workunit.client.0.smithi080.stdout: "agent_state": {} 2024-03-25T00:23:10.315 INFO:tasks.workunit.client.0.smithi080.stdout:} 2024-03-25T00:23:10.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-03-25T00:23:10.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: grep -q obj250 2024-03-25T00:23:10.434 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-03-25T00:23:10.434 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2024-03-25T00:23:10.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: check=true 2024-03-25T00:23:10.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: test true == true 2024-03-25T00:23:10.590 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-03-25T00:23:10.590 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2024-03-25T00:23:10.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2024-03-25T00:23:10.745 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: check='osd is down' 2024-03-25T00:23:10.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2024-03-25T00:23:10.746 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-03-25T00:23:10.746 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2024-03-25T00:23:10.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2024-03-25T00:23:10.899 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: check='2(4)' 2024-03-25T00:23:10.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:581: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:23:10.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:23:10.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:23:10.901 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:23:10.901 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:23:10.901 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:23:10.901 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:23:10.901 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:23:10.901 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:23:10.901 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:23:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-25T00:23:10.903 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:23:10.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:23:10.904 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-03-25T00:23:10.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-25T00:23:10.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:23:10.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:23:10.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:23:10.926 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:10.929+0000 7f41cef177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:23:10.927 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:10.933+0000 7f41cef177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:23:10.929 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:10.933+0000 7f41cef177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:23:11.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-25T00:23:11.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:23:11.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:23:11.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:23:11.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:23:11.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:23:11.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:23:11.288 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:23:11.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:23:11.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:23:11.484 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:11.489+0000 7f41cef177c0 -1 Falling back to public interface 2024-03-25T00:23:11.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:23:12.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:23:12.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:23:12.734 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:23:12.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:23:12.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:23:12.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:23:13.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:23:14.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:23:14.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:23:14.156 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:23:14.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:23:14.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:23:14.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:23:14.346 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:14.349+0000 7f41cef177c0 -1 osd.2 64 log_to_monitors true 2024-03-25T00:23:14.636 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:23:15.637 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:23:15.637 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:23:15.638 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:23:15.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:23:15.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:23:15.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:23:16.270 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 69 up_thru 22 down_at 65 last_clean_interval [57,64) [v2:127.0.0.1:6818/215762399,v1:127.0.0.1:6819/215762399] [v2:127.0.0.1:6820/215762399,v1:127.0.0.1:6821/215762399] exists,up 4298834f-324a-49e0-9eb1-e7d2c17bcdfa 2024-03-25T00:23:16.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:23:16.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:23:16.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:23:16.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-03-25T00:23:21.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:585: TEST_ec_backfill_unfound: test 124 = 124 2024-03-25T00:23:21.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-03-25T00:23:21.560 INFO:tasks.workunit.client.0.smithi080.stderr:pg has no unfound objects 2024-03-25T00:23:21.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:589: TEST_ec_backfill_unfound: wait_for_clean 2024-03-25T00:23:21.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:23:21.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:23:21.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:23:21.579 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:23:21.579 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:23:21.579 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:23:21.579 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:23:21.579 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:23:21.579 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:23:21.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:23:21.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:23:21.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:23:21.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:23:21.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:23:21.688 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:23:22.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:23:22.127 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:23:22.127 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:23:22.127 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:23:22.127 INFO:tasks.workunit.client.0.smithi080.stderr:4' 2024-03-25T00:23:22.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:23:22.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:23:22.127 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:23:22.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803838 2024-03-25T00:23:22.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803838 2024-03-25T00:23:22.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838' 2024-03-25T00:23:22.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:23:22.454 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:23:22.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542204 2024-03-25T00:23:22.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542204 2024-03-25T00:23:22.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-60129542204' 2024-03-25T00:23:22.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:23:22.613 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:23:22.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743428 2024-03-25T00:23:22.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743428 2024-03-25T00:23:22.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-60129542204 2-296352743428' 2024-03-25T00:23:22.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:23:22.798 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:23:22.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=133143986224 2024-03-25T00:23:23.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 133143986224 2024-03-25T00:23:23.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-60129542204 2-296352743428 3-133143986224' 2024-03-25T00:23:23.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:23:23.010 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:23:23.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659173 2024-03-25T00:23:23.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659173 2024-03-25T00:23:23.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-60129542204 2-296352743428 3-133143986224 4-176093659173' 2024-03-25T00:23:23.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:23:23.089 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803838 2024-03-25T00:23:23.089 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:23:23.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:23:23.090 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803838 2024-03-25T00:23:23.090 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:23:23.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803838 2024-03-25T00:23:23.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803838' 2024-03-25T00:23:23.091 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803838 2024-03-25T00:23:23.091 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:23:23.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803836 -lt 25769803838 2024-03-25T00:23:23.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:23:24.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:23:24.494 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:23:25.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803839 -lt 25769803838 2024-03-25T00:23:25.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:23:25.075 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-60129542204 2024-03-25T00:23:25.075 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:23:25.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:23:25.078 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-60129542204 2024-03-25T00:23:25.078 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:23:25.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542204 2024-03-25T00:23:25.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 60129542204' 2024-03-25T00:23:25.079 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 60129542204 2024-03-25T00:23:25.080 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:23:25.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542205 -lt 60129542204 2024-03-25T00:23:25.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:23:25.824 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-296352743428 2024-03-25T00:23:25.824 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:23:25.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:23:25.827 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-296352743428 2024-03-25T00:23:25.827 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:23:25.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743428 2024-03-25T00:23:25.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 296352743428' 2024-03-25T00:23:25.828 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 296352743428 2024-03-25T00:23:25.829 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:23:26.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743428 -lt 296352743428 2024-03-25T00:23:26.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:23:26.250 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-133143986224 2024-03-25T00:23:26.250 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:23:26.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:23:26.252 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-133143986224 2024-03-25T00:23:26.252 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:23:26.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=133143986224 2024-03-25T00:23:26.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 133143986224' 2024-03-25T00:23:26.254 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 133143986224 2024-03-25T00:23:26.254 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:23:26.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 133143986225 -lt 133143986224 2024-03-25T00:23:26.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:23:26.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659173 2024-03-25T00:23:26.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:23:26.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:23:26.686 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659173 2024-03-25T00:23:26.686 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:23:26.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659173 2024-03-25T00:23:26.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659173' 2024-03-25T00:23:26.688 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 176093659173 2024-03-25T00:23:26.688 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:23:27.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659174 -lt 176093659173 2024-03-25T00:23:27.097 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:23:27.098 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:23:27.098 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:23:27.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:23:27.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:23:27.630 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:23:27.630 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:23:27.630 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:23:27.630 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:23:27.630 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:23:27.630 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:23:28.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T00:23:28.042 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:23:28.042 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:23:28.043 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:23:28.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T00:23:28.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T00:23:28.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T00:23:28.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T00:23:28.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T00:23:28.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T00:23:28.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T00:23:28.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:23:28.682 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:23:28.682 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:23:28.682 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:23:28.683 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:23:28.683 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:23:28.683 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:23:29.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T00:23:29.076 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:23:29.076 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:23:29.076 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:23:29.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T00:23:29.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T00:23:29.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T00:23:29.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T00:23:29.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T00:23:29.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T00:23:29.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T00:23:29.608 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T00:23:29.608 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T00:23:30.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12798 2024-03-25T00:23:30.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12798 '!=' null 2024-03-25T00:23:30.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-25T00:23:30.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T00:23:30.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T00:23:30.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T00:23:30.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:23:30.224 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:23:30.224 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:23:30.224 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:23:30.224 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:23:30.225 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:23:30.225 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:23:30.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T00:23:30.607 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:23:30.607 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:23:30.608 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:23:31.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T00:23:31.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T00:23:31.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T00:23:31.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T00:23:31.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T00:23:31.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T00:23:31.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T00:23:31.175 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T00:23:31.175 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T00:23:31.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12799 2024-03-25T00:23:31.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12799 '!=' null 2024-03-25T00:23:31.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-25T00:23:31.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T00:23:31.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T00:23:31.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T00:23:31.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:23:31.861 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:23:31.861 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:23:31.861 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:23:31.861 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:23:31.862 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:23:31.862 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:23:32.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:23:32.285 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:23:32.285 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:23:32.285 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:23:32.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:23:32.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:23:32.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:23:32.796 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: seq 1 300 2024-03-25T00:23:32.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:32.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2024-03-25T00:23:32.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-03-25T00:23:32.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:32.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:32.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2024-03-25T00:23:32.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-03-25T00:23:32.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:32.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:32.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2024-03-25T00:23:32.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-03-25T00:23:32.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:32.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:32.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2024-03-25T00:23:32.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-03-25T00:23:32.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:32.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:32.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2024-03-25T00:23:32.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2024-03-25T00:23:33.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2024-03-25T00:23:33.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2024-03-25T00:23:33.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2024-03-25T00:23:33.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2024-03-25T00:23:33.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2024-03-25T00:23:33.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2024-03-25T00:23:33.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2024-03-25T00:23:33.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.240 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2024-03-25T00:23:33.240 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2024-03-25T00:23:33.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2024-03-25T00:23:33.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2024-03-25T00:23:33.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2024-03-25T00:23:33.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2024-03-25T00:23:33.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2024-03-25T00:23:33.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2024-03-25T00:23:33.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2024-03-25T00:23:33.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2024-03-25T00:23:33.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2024-03-25T00:23:33.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2024-03-25T00:23:33.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2024-03-25T00:23:33.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2024-03-25T00:23:33.617 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2024-03-25T00:23:33.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2024-03-25T00:23:33.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.703 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2024-03-25T00:23:33.703 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2024-03-25T00:23:33.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2024-03-25T00:23:33.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2024-03-25T00:23:33.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2024-03-25T00:23:33.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2024-03-25T00:23:33.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2024-03-25T00:23:33.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2024-03-25T00:23:33.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2024-03-25T00:23:33.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2024-03-25T00:23:33.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-03-25T00:23:33.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:33.990 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:33.990 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2024-03-25T00:23:33.990 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2024-03-25T00:23:34.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2024-03-25T00:23:34.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2024-03-25T00:23:34.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2024-03-25T00:23:34.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2024-03-25T00:23:34.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2024-03-25T00:23:34.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2024-03-25T00:23:34.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2024-03-25T00:23:34.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2024-03-25T00:23:34.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2024-03-25T00:23:34.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2024-03-25T00:23:34.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2024-03-25T00:23:34.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2024-03-25T00:23:34.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2024-03-25T00:23:34.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2024-03-25T00:23:34.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2024-03-25T00:23:34.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2024-03-25T00:23:34.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2024-03-25T00:23:34.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2024-03-25T00:23:34.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2024-03-25T00:23:34.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2024-03-25T00:23:34.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2024-03-25T00:23:34.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2024-03-25T00:23:34.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2024-03-25T00:23:34.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2024-03-25T00:23:34.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2024-03-25T00:23:34.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2024-03-25T00:23:34.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2024-03-25T00:23:34.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2024-03-25T00:23:34.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2024-03-25T00:23:34.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2024-03-25T00:23:34.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2024-03-25T00:23:34.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-03-25T00:23:34.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:34.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:34.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2024-03-25T00:23:34.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2024-03-25T00:23:35.006 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2024-03-25T00:23:35.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2024-03-25T00:23:35.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2024-03-25T00:23:35.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2024-03-25T00:23:35.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2024-03-25T00:23:35.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2024-03-25T00:23:35.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2024-03-25T00:23:35.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.234 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2024-03-25T00:23:35.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2024-03-25T00:23:35.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2024-03-25T00:23:35.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2024-03-25T00:23:35.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.350 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2024-03-25T00:23:35.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2024-03-25T00:23:35.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2024-03-25T00:23:35.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.437 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.437 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.437 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2024-03-25T00:23:35.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2024-03-25T00:23:35.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2024-03-25T00:23:35.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2024-03-25T00:23:35.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2024-03-25T00:23:35.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2024-03-25T00:23:35.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2024-03-25T00:23:35.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2024-03-25T00:23:35.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2024-03-25T00:23:35.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2024-03-25T00:23:35.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2024-03-25T00:23:35.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2024-03-25T00:23:35.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2024-03-25T00:23:35.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2024-03-25T00:23:35.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2024-03-25T00:23:35.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2024-03-25T00:23:35.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2024-03-25T00:23:35.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2024-03-25T00:23:35.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2024-03-25T00:23:35.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2024-03-25T00:23:35.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:35.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:35.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2024-03-25T00:23:35.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2024-03-25T00:23:36.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2024-03-25T00:23:36.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2024-03-25T00:23:36.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2024-03-25T00:23:36.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2024-03-25T00:23:36.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2024-03-25T00:23:36.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2024-03-25T00:23:36.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2024-03-25T00:23:36.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2024-03-25T00:23:36.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.285 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.285 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2024-03-25T00:23:36.285 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2024-03-25T00:23:36.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2024-03-25T00:23:36.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2024-03-25T00:23:36.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2024-03-25T00:23:36.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2024-03-25T00:23:36.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2024-03-25T00:23:36.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2024-03-25T00:23:36.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2024-03-25T00:23:36.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2024-03-25T00:23:36.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2024-03-25T00:23:36.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2024-03-25T00:23:36.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2024-03-25T00:23:36.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.658 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2024-03-25T00:23:36.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2024-03-25T00:23:36.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.770 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.770 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2024-03-25T00:23:36.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2024-03-25T00:23:36.800 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2024-03-25T00:23:36.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2024-03-25T00:23:36.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2024-03-25T00:23:36.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2024-03-25T00:23:36.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2024-03-25T00:23:36.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2024-03-25T00:23:36.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:36.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:36.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2024-03-25T00:23:36.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2024-03-25T00:23:37.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2024-03-25T00:23:37.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2024-03-25T00:23:37.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2024-03-25T00:23:37.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.116 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2024-03-25T00:23:37.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2024-03-25T00:23:37.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2024-03-25T00:23:37.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2024-03-25T00:23:37.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.232 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2024-03-25T00:23:37.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.260 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2024-03-25T00:23:37.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2024-03-25T00:23:37.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2024-03-25T00:23:37.320 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.350 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.350 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2024-03-25T00:23:37.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2024-03-25T00:23:37.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2024-03-25T00:23:37.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2024-03-25T00:23:37.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2024-03-25T00:23:37.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2024-03-25T00:23:37.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2024-03-25T00:23:37.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.548 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2024-03-25T00:23:37.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2024-03-25T00:23:37.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2024-03-25T00:23:37.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2024-03-25T00:23:37.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2024-03-25T00:23:37.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2024-03-25T00:23:37.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2024-03-25T00:23:37.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2024-03-25T00:23:37.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2024-03-25T00:23:37.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.807 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2024-03-25T00:23:37.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.836 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2024-03-25T00:23:37.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2024-03-25T00:23:37.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2024-03-25T00:23:37.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2024-03-25T00:23:37.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2024-03-25T00:23:37.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2024-03-25T00:23:37.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:37.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:37.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2024-03-25T00:23:37.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2024-03-25T00:23:38.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2024-03-25T00:23:38.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2024-03-25T00:23:38.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2024-03-25T00:23:38.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2024-03-25T00:23:38.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2024-03-25T00:23:38.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2024-03-25T00:23:38.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2024-03-25T00:23:38.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2024-03-25T00:23:38.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2024-03-25T00:23:38.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.315 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2024-03-25T00:23:38.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2024-03-25T00:23:38.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2024-03-25T00:23:38.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2024-03-25T00:23:38.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2024-03-25T00:23:38.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2024-03-25T00:23:38.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2024-03-25T00:23:38.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2024-03-25T00:23:38.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2024-03-25T00:23:38.548 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2024-03-25T00:23:38.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2024-03-25T00:23:38.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.633 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2024-03-25T00:23:38.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2024-03-25T00:23:38.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2024-03-25T00:23:38.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2024-03-25T00:23:38.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2024-03-25T00:23:38.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.800 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.801 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.801 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2024-03-25T00:23:38.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2024-03-25T00:23:38.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2024-03-25T00:23:38.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2024-03-25T00:23:38.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2024-03-25T00:23:38.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2024-03-25T00:23:38.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2024-03-25T00:23:38.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:38.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:38.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2024-03-25T00:23:38.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2024-03-25T00:23:39.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2024-03-25T00:23:39.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.074 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2024-03-25T00:23:39.074 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.102 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.102 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2024-03-25T00:23:39.102 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.131 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.131 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2024-03-25T00:23:39.131 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2024-03-25T00:23:39.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2024-03-25T00:23:39.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2024-03-25T00:23:39.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2024-03-25T00:23:39.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2024-03-25T00:23:39.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2024-03-25T00:23:39.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2024-03-25T00:23:39.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2024-03-25T00:23:39.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2024-03-25T00:23:39.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.419 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.419 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.419 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2024-03-25T00:23:39.419 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2024-03-25T00:23:39.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2024-03-25T00:23:39.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2024-03-25T00:23:39.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2024-03-25T00:23:39.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2024-03-25T00:23:39.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2024-03-25T00:23:39.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2024-03-25T00:23:39.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2024-03-25T00:23:39.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2024-03-25T00:23:39.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2024-03-25T00:23:39.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2024-03-25T00:23:39.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2024-03-25T00:23:39.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2024-03-25T00:23:39.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2024-03-25T00:23:39.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2024-03-25T00:23:39.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2024-03-25T00:23:39.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2024-03-25T00:23:39.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2024-03-25T00:23:39.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2024-03-25T00:23:39.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2024-03-25T00:23:39.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:39.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:39.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2024-03-25T00:23:39.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2024-03-25T00:23:40.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2024-03-25T00:23:40.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2024-03-25T00:23:40.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2024-03-25T00:23:40.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2024-03-25T00:23:40.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.168 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.168 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2024-03-25T00:23:40.168 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2024-03-25T00:23:40.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.236 INFO:tasks.workunit.client.0.smithi080.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2024-03-25T00:23:40.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2024-03-25T00:23:40.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2024-03-25T00:23:40.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2024-03-25T00:23:40.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2024-03-25T00:23:40.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2024-03-25T00:23:40.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2024-03-25T00:23:40.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2024-03-25T00:23:40.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2024-03-25T00:23:40.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2024-03-25T00:23:40.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2024-03-25T00:23:40.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2024-03-25T00:23:40.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2024-03-25T00:23:40.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2024-03-25T00:23:40.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2024-03-25T00:23:40.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2024-03-25T00:23:40.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2024-03-25T00:23:40.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2024-03-25T00:23:40.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2024-03-25T00:23:40.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.848 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.848 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.848 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2024-03-25T00:23:40.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2024-03-25T00:23:40.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2024-03-25T00:23:40.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2024-03-25T00:23:40.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2024-03-25T00:23:40.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2024-03-25T00:23:40.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:40.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:40.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2024-03-25T00:23:40.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2024-03-25T00:23:41.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2024-03-25T00:23:41.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2024-03-25T00:23:41.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2024-03-25T00:23:41.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2024-03-25T00:23:41.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2024-03-25T00:23:41.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2024-03-25T00:23:41.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2024-03-25T00:23:41.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2024-03-25T00:23:41.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2024-03-25T00:23:41.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2024-03-25T00:23:41.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2024-03-25T00:23:41.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2024-03-25T00:23:41.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2024-03-25T00:23:41.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2024-03-25T00:23:41.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2024-03-25T00:23:41.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2024-03-25T00:23:41.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2024-03-25T00:23:41.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.539 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2024-03-25T00:23:41.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2024-03-25T00:23:41.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2024-03-25T00:23:41.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.627 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2024-03-25T00:23:41.627 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2024-03-25T00:23:41.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2024-03-25T00:23:41.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2024-03-25T00:23:41.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:23:41.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2024-03-25T00:23:41.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2024-03-25T00:23:41.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:602: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:23:41.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2024-03-25T00:23:41.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:23:41.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-03-25T00:23:42.247 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' does not exist 2024-03-25T00:23:42.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-03-25T00:23:43.039 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile myprofile does not exist 2024-03-25T00:23:43.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-03-25T00:23:43.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:23:43.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:23:43.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:23:43.060 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:23:43.060 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:23:43.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:23:43.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:23:43.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:23:43.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:23:43.374 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:23:43.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:23:43.375 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:23:43.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:23:43.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:23:43.377 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:23:43.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:23:43.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:23:43.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:23:43.379 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:23:43.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:23:43.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:23:43.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:23:43.444 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:23:43.445 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:23:43.445 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:23:43.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:23:43.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:23:43.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:23:43.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:23:43.450 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:23:43.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:23:43.451 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:23:43.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:23:43.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:23:43.452 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:23:43.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:23:43.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:23:43.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:23:43.454 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:23:43.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:23:43.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:23:43.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:23:43.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:23:43.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:23:43.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:23:43.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:23:43.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:23:43.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:23:43.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-03-25T00:23:43.460 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:23:43.460 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:23:43.460 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:23:43.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81310 2024-03-25T00:23:43.462 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:23:43.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:23:43.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:23:43.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-03-25T00:23:43.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-03-25T00:23:43.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-03-25T00:23:43.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:23:43.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:23:43.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:23:43.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-03-25T00:23:43.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-03-25T00:23:43.884 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:23:43.884 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:23:43.885 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:23:43.885 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:23:43.885 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:23:43.885 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:23:43.885 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:23:43.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T00:23:44.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:23:44.069 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:23:44.069 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:23:44.069 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:23:44.069 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:23:44.070 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:23:44.071 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:23:44.071 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:23:44.071 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:23:44.071 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:23:44.071 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:23:44.071 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:23:44.071 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:23:44.071 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:23:44.071 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get fsid 2024-03-25T00:23:44.187 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:23:44.187 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:23:44.187 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:23:44.187 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:23:44.187 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:23:44.188 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:23:44.188 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:23:44.188 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:23:44.188 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:23:44.189 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:23:44.189 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:23:44.189 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:23:44.189 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:23:44.189 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get mon_host 2024-03-25T00:23:44.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-03-25T00:23:44.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-03-25T00:23:44.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:23:44.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:23:44.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:23:44.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-03-25T00:23:44.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:23:45.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:23:45.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:23:45.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:23:45.213 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:23:45.213 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:23:45.213 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:23:45.213 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:23:45.214 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:23:45.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:23:45.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-03-25T00:23:45.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T00:23:45.515 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' already exists 2024-03-25T00:23:45.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:23:46.536 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-03-25T00:23:46.536 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:23:46.536 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:23:46.537 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:23:46.537 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:23:46.537 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:23:46.537 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:23:46.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-03-25T00:23:46.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-mon.a.asok log flush 2024-03-25T00:23:46.642 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:23:46.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-03-25T00:23:46.656 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:23:43.936+0000 7f603f033c80 0 load: jerasure load: lrc 2024-03-25T00:23:46.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2024-03-25T00:23:46.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:359: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2024-03-25T00:23:46.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_ec_object_attr_read_error: local objname=myobject 2024-03-25T00:23:46.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:362: TEST_ec_object_attr_read_error: setup_osds 7 2024-03-25T00:23:46.658 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-03-25T00:23:46.658 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-03-25T00:23:46.658 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-03-25T00:23:46.659 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-03-25T00:23:46.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:23:46.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-03-25T00:23:46.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:23:46.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:23:46.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:23:46.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:23:46.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-03-25T00:23:46.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:23:46.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:23:46.663 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:23:46.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:23:46.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:23:46.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:23:46.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:23:46.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:23:46.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:23:46.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:23:46.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:23:46.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:23:46.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:23:46.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:23:46.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:23:46.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-03-25T00:23:46.666 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:23:46.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a70f8cf9-c2ae-404e-8cd7-721c303ebf10 2024-03-25T00:23:46.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a70f8cf9-c2ae-404e-8cd7-721c303ebf10' 2024-03-25T00:23:46.667 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 a70f8cf9-c2ae-404e-8cd7-721c303ebf10 2024-03-25T00:23:46.667 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:23:46.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASxABmX7o7KRAAUh37s502Kww3w1E6ZQJTrA== 2024-03-25T00:23:46.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASxABmX7o7KRAAUh37s502Kww3w1E6ZQJTrA=="}' 2024-03-25T00:23:46.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a70f8cf9-c2ae-404e-8cd7-721c303ebf10 -i td/test-erasure-eio/0/new.json 2024-03-25T00:23:46.983 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:23:46.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-03-25T00:23:46.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASxABmX7o7KRAAUh37s502Kww3w1E6ZQJTrA== --osd-uuid a70f8cf9-c2ae-404e-8cd7-721c303ebf10 2024-03-25T00:23:47.027 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:47.036+0000 7fdd27b547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:23:47.043 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:47.052+0000 7fdd27b547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:23:47.044 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:47.052+0000 7fdd27b547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:23:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:47.052+0000 7fdd27b547c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:23:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:47.052+0000 7fdd27b547c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:23:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:47.052+0000 7fdd27b547c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:23:47.083 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:47.092+0000 7fdd27b547c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-03-25T00:23:52.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-03-25T00:23:52.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:23:52.703 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:23:52.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:23:52.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:23:53.251 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:23:53.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:23:53.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:23:53.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:23:53.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:23:53.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:23:53.282 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:53.288+0000 7f2464f9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:23:53.284 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:53.292+0000 7f2464f9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:23:53.285 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:53.292+0000 7f2464f9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:23:53.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:23:53.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:23:53.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:23:53.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:23:53.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:23:53.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:23:53.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:23:53.697 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:23:53.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:23:53.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:23:53.864 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:53.872+0000 7f2464f9f7c0 -1 Falling back to public interface 2024-03-25T00:23:54.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:23:55.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:23:55.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:23:55.099 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:23:55.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:23:55.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:23:55.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:23:55.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:23:56.406 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:23:56.412+0000 7f2464f9f7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:23:56.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:23:56.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:23:56.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:23:56.508 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:23:56.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:23:56.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:23:57.007 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:23:58.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:23:58.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:23:58.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:23:58.009 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:23:58.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:23:58.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:23:58.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:23:59.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:23:59.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:23:59.471 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:23:59.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:23:59.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:23:59.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:23:59.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:00.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:00.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:00.885 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:24:00.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:24:00.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:00.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:01.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:02.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:02.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:02.304 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:24:02.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:24:02.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:02.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:02.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:03.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:03.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:03.741 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:24:03.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:24:03.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:03.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:04.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:05.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:05.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:05.245 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:24:05.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:24:05.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:05.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:05.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:06.703 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:06.703 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:06.704 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:24:06.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:24:06.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:06.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:07.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:08.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:08.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:08.267 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:24:08.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:24:08.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:08.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:08.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:09.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:09.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:09.897 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:24:09.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:24:09.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:09.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:10.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:11.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:11.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:11.401 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:24:11.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:24:11.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:11.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:12.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:13.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:13.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:13.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:24:13.291 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:24:13.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:13.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:14.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:15.011 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:15.011 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:15.011 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:24:15.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:24:15.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:15.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:15.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:16.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:16.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:16.892 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:24:16.893 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:24:16.893 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:16.893 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:17.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:18.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:18.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:18.827 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:24:18.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:24:18.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:18.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:24:19.300 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3695755489,v1:127.0.0.1:6803/3695755489] [v2:127.0.0.1:6804/3695755489,v1:127.0.0.1:6805/3695755489] exists,up a70f8cf9-c2ae-404e-8cd7-721c303ebf10 2024-03-25T00:24:19.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:24:19.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:24:19.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:24:19.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:24:19.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:24:19.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:24:19.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:24:19.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:24:19.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:24:19.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:24:19.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:24:19.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:24:19.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:24:19.310 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:24:19.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a1d33b14-e906-4e92-92b9-62ce9ca6d226 2024-03-25T00:24:19.310 INFO:tasks.workunit.client.0.smithi080.stdout:add osd1 a1d33b14-e906-4e92-92b9-62ce9ca6d226 2024-03-25T00:24:19.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a1d33b14-e906-4e92-92b9-62ce9ca6d226' 2024-03-25T00:24:19.311 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:24:19.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAzxABm3IwhFBAAm3ti+HqeWeCk+aySsUAlwQ== 2024-03-25T00:24:19.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAzxABm3IwhFBAAm3ti+HqeWeCk+aySsUAlwQ=="}' 2024-03-25T00:24:19.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a1d33b14-e906-4e92-92b9-62ce9ca6d226 -i td/test-erasure-eio/1/new.json 2024-03-25T00:24:19.806 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:24:19.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-03-25T00:24:19.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzxABm3IwhFBAAm3ti+HqeWeCk+aySsUAlwQ== --osd-uuid a1d33b14-e906-4e92-92b9-62ce9ca6d226 2024-03-25T00:24:19.856 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:19.863+0000 7f7246f147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:24:19.874 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:19.879+0000 7f7246f147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:24:19.876 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:19.883+0000 7f7246f147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:24:19.876 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:19.883+0000 7f7246f147c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:24:19.876 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:19.883+0000 7f7246f147c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:24:19.876 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:19.883+0000 7f7246f147c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:24:19.974 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:19.979+0000 7f7246f147c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-03-25T00:24:25.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-03-25T00:24:25.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:24:25.546 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd1 key to auth repository 2024-03-25T00:24:25.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T00:24:25.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:24:26.089 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.1 2024-03-25T00:24:26.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T00:24:26.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:24:26.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:24:26.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:24:26.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:24:26.120 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:26.127+0000 7f5c3cdb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:24:26.138 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:26.143+0000 7f5c3cdb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:24:26.139 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:26.143+0000 7f5c3cdb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:24:26.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T00:24:26.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:24:26.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:24:26.535 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:24:26.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:24:26.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:24:26.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:26.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:24:26.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:26.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:26.720 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:26.727+0000 7f5c3cdb27c0 -1 Falling back to public interface 2024-03-25T00:24:26.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:27.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:27.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:27.987 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:24:27.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:24:27.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:27.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:28.437 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:29.230 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:29.235+0000 7f5c3cdb27c0 -1 osd.1 0 log_to_monitors true 2024-03-25T00:24:29.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:29.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:29.440 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:24:29.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:24:29.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:29.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:30.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:31.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:31.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:31.057 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:24:31.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:24:31.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:31.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:31.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:32.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:32.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:32.472 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:24:32.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:24:32.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:32.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:33.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:34.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:34.011 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:34.011 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:24:34.011 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:24:34.011 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:34.011 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:34.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:35.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:35.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:35.599 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:24:35.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:24:35.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:35.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:36.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:37.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:37.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:37.125 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:24:37.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:24:37.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:37.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:37.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:38.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:38.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:38.580 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:24:38.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:24:38.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:38.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:39.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:40.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:40.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:40.087 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:24:40.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:24:40.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:40.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:40.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:41.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:41.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:41.572 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:24:41.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:24:41.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:41.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:42.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:43.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:43.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:43.053 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:24:43.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:24:43.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:43.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:44.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:45.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:45.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:45.108 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:24:45.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:24:45.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:45.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:45.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:46.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:46.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:46.882 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:24:46.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:24:46.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:46.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:47.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:48.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:48.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:48.708 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:24:48.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:24:48.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:48.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:49.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:50.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:50.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:50.426 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:24:50.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:24:50.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:50.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:50.896 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:51.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:51.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:51.898 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:24:51.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:24:51.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:51.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:52.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:24:53.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:24:53.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:24:53.507 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:24:53.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:24:53.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:24:53.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:24:53.969 INFO:tasks.workunit.client.0.smithi080.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1667751641,v1:127.0.0.1:6811/1667751641] [v2:127.0.0.1:6812/1667751641,v1:127.0.0.1:6813/1667751641] exists,up a1d33b14-e906-4e92-92b9-62ce9ca6d226 2024-03-25T00:24:53.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:24:53.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:24:53.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:24:53.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:24:53.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:24:53.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:24:53.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:24:53.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:24:53.973 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:24:53.973 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:24:53.973 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:24:53.973 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:24:53.973 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:24:53.973 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:24:53.973 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:24:53.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:24:53.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:24:53.976 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:24:53.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=12103197-3927-402f-99ec-ae6c6ff5d97d 2024-03-25T00:24:53.977 INFO:tasks.workunit.client.0.smithi080.stdout:add osd2 12103197-3927-402f-99ec-ae6c6ff5d97d 2024-03-25T00:24:53.977 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 12103197-3927-402f-99ec-ae6c6ff5d97d' 2024-03-25T00:24:53.977 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:24:53.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVxABmde6MOxAAULu+kGSeQu7dXX52ad7IXA== 2024-03-25T00:24:53.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVxABmde6MOxAAULu+kGSeQu7dXX52ad7IXA=="}' 2024-03-25T00:24:53.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 12103197-3927-402f-99ec-ae6c6ff5d97d -i td/test-erasure-eio/2/new.json 2024-03-25T00:24:54.501 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:24:54.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-03-25T00:24:54.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVxABmde6MOxAAULu+kGSeQu7dXX52ad7IXA== --osd-uuid 12103197-3927-402f-99ec-ae6c6ff5d97d 2024-03-25T00:24:54.555 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:54.558+0000 7fee54f497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:24:54.588 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:54.594+0000 7fee54f497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:24:54.590 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:54.594+0000 7fee54f497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:24:54.591 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:54.594+0000 7fee54f497c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:24:54.591 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:54.594+0000 7fee54f497c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:24:54.591 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:54.594+0000 7fee54f497c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:24:54.626 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:24:54.630+0000 7fee54f497c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-03-25T00:25:00.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-03-25T00:25:00.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:25:00.196 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd2 key to auth repository 2024-03-25T00:25:00.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T00:25:00.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:25:00.771 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:25:00.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T00:25:00.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:25:00.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:25:00.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:25:00.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:25:00.802 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:00.806+0000 7fc7417d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:25:00.804 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:00.810+0000 7fc7417d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:25:00.805 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:00.810+0000 7fc7417d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:25:01.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T00:25:01.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:25:01.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:25:01.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:25:01.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:25:01.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:01.206 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:25:01.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:25:01.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:01.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:01.358 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:01.362+0000 7fc7417d67c0 -1 Falling back to public interface 2024-03-25T00:25:01.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:02.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:02.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:02.646 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:25:02.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:25:02.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:02.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:03.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:03.949 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:03.954+0000 7fc7417d67c0 -1 osd.2 0 log_to_monitors true 2024-03-25T00:25:04.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:04.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:04.049 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:25:04.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:25:04.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:04.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:04.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:05.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:05.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:05.687 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:25:05.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:25:05.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:05.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:06.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:07.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:07.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:07.127 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:25:07.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:25:07.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:07.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:07.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:08.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:08.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:08.584 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:25:08.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:25:08.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:08.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:09.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:10.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:10.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:10.107 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:25:10.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:25:10.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:10.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:10.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:11.628 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:11.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:11.629 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:25:11.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:25:11.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:11.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:12.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:13.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:13.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:13.152 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:25:13.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:25:13.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:13.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:13.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:14.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:14.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:14.578 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:25:14.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:25:14.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:14.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:15.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:16.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:16.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:16.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:25:16.021 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:25:16.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:16.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:16.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:17.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:17.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:17.491 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:25:17.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:25:17.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:17.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:18.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:19.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:19.522 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:25:19.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:19.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:25:19.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:19.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:20.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:21.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:21.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:21.163 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:25:21.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:25:21.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:21.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:22.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:23.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:23.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:23.068 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:25:23.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:25:23.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:23.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:23.777 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:24.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:24.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:24.779 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:25:24.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:25:24.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:24.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:25.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:26.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:26.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:26.776 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:25:26.776 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:25:26.776 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:26.776 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:25:27.325 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3003029234,v1:127.0.0.1:6819/3003029234] [v2:127.0.0.1:6820/3003029234,v1:127.0.0.1:6821/3003029234] exists,up 12103197-3927-402f-99ec-ae6c6ff5d97d 2024-03-25T00:25:27.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:25:27.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:25:27.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:25:27.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:25:27.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-03-25T00:25:27.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:25:27.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:25:27.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T00:25:27.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:25:27.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:25:27.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:25:27.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:25:27.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:25:27.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:25:27.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:25:27.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:25:27.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:25:27.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:25:27.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:25:27.330 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:25:27.330 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:25:27.330 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:25:27.330 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:25:27.330 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:25:27.330 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:25:27.330 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:25:27.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:25:27.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:25:27.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:25:27.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:25:27.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:25:27.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:25:27.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:25:27.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:25:27.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:25:27.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:25:27.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:25:27.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:25:27.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:25:27.332 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:25:27.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=05ec8d40-942e-42b8-9a5a-7d205ccdd13d 2024-03-25T00:25:27.332 INFO:tasks.workunit.client.0.smithi080.stdout:add osd3 05ec8d40-942e-42b8-9a5a-7d205ccdd13d 2024-03-25T00:25:27.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 05ec8d40-942e-42b8-9a5a-7d205ccdd13d' 2024-03-25T00:25:27.333 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:25:27.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3xABmjddWFRAA9cuV2Bc/HdeqnRedAEfH6g== 2024-03-25T00:25:27.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3xABmjddWFRAA9cuV2Bc/HdeqnRedAEfH6g=="}' 2024-03-25T00:25:27.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 05ec8d40-942e-42b8-9a5a-7d205ccdd13d -i td/test-erasure-eio/3/new.json 2024-03-25T00:25:28.136 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:25:28.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-03-25T00:25:28.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3xABmjddWFRAA9cuV2Bc/HdeqnRedAEfH6g== --osd-uuid 05ec8d40-942e-42b8-9a5a-7d205ccdd13d 2024-03-25T00:25:28.193 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:28.197+0000 7f1693e227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:25:28.196 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:28.197+0000 7f1693e227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:25:28.197 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:28.201+0000 7f1693e227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:25:28.198 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:28.201+0000 7f1693e227c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:25:28.198 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:28.201+0000 7f1693e227c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:25:28.198 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:28.201+0000 7f1693e227c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:25:28.321 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:28.325+0000 7f1693e227c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-03-25T00:25:34.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-03-25T00:25:34.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:25:34.142 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd3 key to auth repository 2024-03-25T00:25:34.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T00:25:34.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:25:34.696 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.3 2024-03-25T00:25:34.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T00:25:34.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:25:34.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:25:34.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:25:34.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:25:34.727 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:34.732+0000 7f86fe9817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:25:34.729 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:34.736+0000 7f86fe9817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:25:34.730 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:34.736+0000 7f86fe9817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:25:35.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T00:25:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:25:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:25:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:25:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:25:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:25:35.140 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:25:35.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:35.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:35.306 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:35.312+0000 7f86fe9817c0 -1 Falling back to public interface 2024-03-25T00:25:35.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:36.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:36.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:36.595 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:25:36.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:25:36.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:36.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:37.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:37.940 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:25:37.944+0000 7f86fe9817c0 -1 osd.3 0 log_to_monitors true 2024-03-25T00:25:38.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:38.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:38.006 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:25:38.006 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:25:38.006 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:38.006 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:38.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:39.658 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:39.658 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:39.658 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:25:39.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:25:39.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:39.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:40.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:41.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:41.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:41.314 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:25:41.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:25:41.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:41.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:41.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:42.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:42.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:42.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:25:42.774 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:25:42.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:42.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:43.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:44.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:44.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:44.235 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:25:44.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:25:44.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:44.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:44.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:45.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:45.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:45.728 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:25:45.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:25:45.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:45.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:46.131 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:47.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:47.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:47.133 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:25:47.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:25:47.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:47.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:47.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:48.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:48.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:48.545 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:25:48.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:25:48.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:48.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:48.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:49.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:49.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:49.964 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:25:49.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:25:49.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:49.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:50.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:51.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:51.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:51.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:25:51.559 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:25:51.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:51.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:52.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:53.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:53.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:53.156 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:25:53.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:25:53.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:53.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:53.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:54.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:54.899 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:25:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:25:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:55.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:56.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:56.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:56.573 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:25:56.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:25:56.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:56.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:57.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:25:58.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:25:58.663 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:25:58.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:25:58.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:25:58.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:25:58.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:25:59.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:00.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:00.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:00.558 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:26:00.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:26:00.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:00.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:26:01.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:02.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:02.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:02.067 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:26:02.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:26:02.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:02.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:26:02.585 INFO:tasks.workunit.client.0.smithi080.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3132709409,v1:127.0.0.1:6827/3132709409] [v2:127.0.0.1:6828/3132709409,v1:127.0.0.1:6829/3132709409] exists,up 05ec8d40-942e-42b8-9a5a-7d205ccdd13d 2024-03-25T00:26:02.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:26:02.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:26:02.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:26:02.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:26:02.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-03-25T00:26:02.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:26:02.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:26:02.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-25T00:26:02.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:26:02.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-03-25T00:26:02.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:26:02.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:26:02.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:26:02.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:26:02.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-03-25T00:26:02.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-03-25T00:26:02.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:26:02.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:26:02.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:26:02.592 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:26:02.592 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:26:02.592 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:26:02.592 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:26:02.592 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:26:02.592 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:26:02.593 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:26:02.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:26:02.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:26:02.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:26:02.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:26:02.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:26:02.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:26:02.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:26:02.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:26:02.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:26:02.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:26:02.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:26:02.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:26:02.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-03-25T00:26:02.595 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:26:02.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4262e85f-2fef-4f4a-9f90-8933260e6080 2024-03-25T00:26:02.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 4262e85f-2fef-4f4a-9f90-8933260e6080' 2024-03-25T00:26:02.597 INFO:tasks.workunit.client.0.smithi080.stdout:add osd4 4262e85f-2fef-4f4a-9f90-8933260e6080 2024-03-25T00:26:02.597 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:26:02.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCaxABmcfHtJBAAKITdGMYx4QzZMBqpHNdxAQ== 2024-03-25T00:26:02.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCaxABmcfHtJBAAKITdGMYx4QzZMBqpHNdxAQ=="}' 2024-03-25T00:26:02.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4262e85f-2fef-4f4a-9f90-8933260e6080 -i td/test-erasure-eio/4/new.json 2024-03-25T00:26:03.308 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:26:03.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-03-25T00:26:03.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCaxABmcfHtJBAAKITdGMYx4QzZMBqpHNdxAQ== --osd-uuid 4262e85f-2fef-4f4a-9f90-8933260e6080 2024-03-25T00:26:03.352 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:03.356+0000 7fd3920a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:03.354 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:03.360+0000 7fd3920a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:03.355 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:03.360+0000 7fd3920a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:03.356 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:03.360+0000 7fd3920a27c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:26:03.356 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:03.360+0000 7fd3920a27c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:26:03.356 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:03.360+0000 7fd3920a27c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:26:03.476 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:03.480+0000 7fd3920a27c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-03-25T00:26:09.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-03-25T00:26:09.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:26:09.674 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd4 key to auth repository 2024-03-25T00:26:09.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-25T00:26:09.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:26:10.243 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.4 2024-03-25T00:26:10.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-25T00:26:10.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:26:10.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:26:10.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:26:10.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:26:10.274 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:10.279+0000 7f33c62ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:10.312 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:10.315+0000 7f33c62ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:10.313 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:10.319+0000 7f33c62ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:10.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-25T00:26:10.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:26:10.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-25T00:26:10.692 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:26:10.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:26:10.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:26:10.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:10.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:26:10.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:10.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:10.890 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:10.895+0000 7f33c62ba7c0 -1 Falling back to public interface 2024-03-25T00:26:11.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:12.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:12.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:12.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:26:12.178 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:26:12.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:12.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:12.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:13.461 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:13.463+0000 7f33c62ba7c0 -1 osd.4 0 log_to_monitors true 2024-03-25T00:26:13.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:13.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:13.621 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:26:13.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:26:13.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:13.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:14.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:15.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:15.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:15.109 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:26:15.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:26:15.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:15.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:15.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:16.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:16.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:16.598 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:26:16.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:26:16.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:16.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:17.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:18.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:18.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:18.172 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:26:18.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:26:18.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:18.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:18.777 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:19.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:19.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:19.779 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:26:19.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:26:19.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:19.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:20.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:21.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:21.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:21.264 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:26:21.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:26:21.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:21.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:21.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:22.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:22.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:22.695 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:26:22.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:26:22.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:22.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:23.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:24.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:24.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:24.307 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:26:24.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:26:24.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:24.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:24.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:25.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:25.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:25.759 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:26:25.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:26:25.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:25.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:26.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:27.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:27.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:27.327 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:26:27.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:26:27.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:27.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:28.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:29.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:29.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:29.279 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:26:29.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:26:29.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:29.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:30.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:31.398 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:31.398 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:31.398 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:26:31.398 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:26:31.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:31.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:32.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:33.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:33.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:33.218 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:26:33.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:26:33.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:33.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:34.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:35.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:35.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:35.078 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:26:35.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:26:35.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:35.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:35.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:36.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:36.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:36.737 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:26:36.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:26:36.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:36.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:26:37.180 INFO:tasks.workunit.client.0.smithi080.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2609960512,v1:127.0.0.1:6835/2609960512] [v2:127.0.0.1:6836/2609960512,v1:127.0.0.1:6837/2609960512] exists,up 4262e85f-2fef-4f4a-9f90-8933260e6080 2024-03-25T00:26:37.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:26:37.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:26:37.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:26:37.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:26:37.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:26:37.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:26:37.184 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:26:37.184 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:26:37.184 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:26:37.185 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:26:37.185 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:26:37.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:26:37.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-03-25T00:26:37.187 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:26:37.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=26cfbeab-b22a-45e5-8cb7-0acaa9cb6126 2024-03-25T00:26:37.187 INFO:tasks.workunit.client.0.smithi080.stdout:add osd5 26cfbeab-b22a-45e5-8cb7-0acaa9cb6126 2024-03-25T00:26:37.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 26cfbeab-b22a-45e5-8cb7-0acaa9cb6126' 2024-03-25T00:26:37.188 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:26:37.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9xABmRAaVDBAAb24s8WOpqmQNOX2UMJ7U8g== 2024-03-25T00:26:37.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9xABmRAaVDBAAb24s8WOpqmQNOX2UMJ7U8g=="}' 2024-03-25T00:26:37.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 26cfbeab-b22a-45e5-8cb7-0acaa9cb6126 -i td/test-erasure-eio/5/new.json 2024-03-25T00:26:38.095 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:26:38.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-03-25T00:26:38.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9xABmRAaVDBAAb24s8WOpqmQNOX2UMJ7U8g== --osd-uuid 26cfbeab-b22a-45e5-8cb7-0acaa9cb6126 2024-03-25T00:26:38.156 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:38.158+0000 7fb0faa3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:38.158 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:38.162+0000 7fb0faa3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:38.159 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:38.162+0000 7fb0faa3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:38.160 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:38.162+0000 7fb0faa3f7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:26:38.160 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:38.162+0000 7fb0faa3f7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:26:38.160 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:38.162+0000 7fb0faa3f7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:26:38.275 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:38.278+0000 7fb0faa3f7c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-03-25T00:26:46.102 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-03-25T00:26:46.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:26:46.103 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd5 key to auth repository 2024-03-25T00:26:46.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-25T00:26:46.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:26:46.652 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.5 2024-03-25T00:26:46.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-25T00:26:46.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:26:46.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:26:46.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:26:46.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:26:46.683 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:46.686+0000 7f63d72e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:46.688 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:46.690+0000 7f63d72e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:46.689 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:46.690+0000 7f63d72e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:26:47.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-25T00:26:47.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:26:47.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T00:26:47.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:26:47.044 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:26:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:26:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:26:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:26:47.241 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:47.242+0000 7f63d72e77c0 -1 Falling back to public interface 2024-03-25T00:26:47.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:48.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:48.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:48.458 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:26:48.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:26:48.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:48.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:26:48.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:49.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:49.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:49.873 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:26:49.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:26:49.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:49.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:26:49.935 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:26:49.938+0000 7f63d72e77c0 -1 osd.5 0 log_to_monitors true 2024-03-25T00:26:50.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:51.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:51.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:51.304 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:26:51.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:26:51.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:51.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:26:51.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:52.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:52.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:52.861 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:26:52.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:26:52.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:52.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:26:53.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:54.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:54.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:54.444 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:26:54.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:26:54.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:54.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:26:55.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:56.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:56.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:56.041 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:26:56.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:26:56.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:56.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:26:56.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:57.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:57.625 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:57.625 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:26:57.625 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:26:57.625 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:57.625 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:26:58.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:26:59.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:26:59.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:26:59.261 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:26:59.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:26:59.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:26:59.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:26:59.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:00.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:00.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:00.696 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:27:00.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:27:00.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:00.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:27:01.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:02.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:02.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:02.211 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:27:02.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:27:02.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:02.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:27:02.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:03.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:03.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:03.699 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:27:03.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:27:03.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:03.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:27:04.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:05.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:05.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:05.316 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:27:05.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:27:05.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:05.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:27:05.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:06.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:06.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:06.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:27:06.939 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:27:06.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:06.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:27:07.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:08.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:08.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:08.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:27:08.901 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:27:08.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:08.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:27:09.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:10.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:27:10.902 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:27:10.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:10.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:27:11.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:12.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:12.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:12.882 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:27:12.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:27:12.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:12.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:27:13.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:14.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:14.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:14.895 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:27:14.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:27:14.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:14.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:27:15.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:16.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:16.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:16.740 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:27:16.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:27:16.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:16.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:27:17.218 INFO:tasks.workunit.client.0.smithi080.stdout:osd.5 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2515290215,v1:127.0.0.1:6843/2515290215] [v2:127.0.0.1:6844/2515290215,v1:127.0.0.1:6845/2515290215] exists,up 26cfbeab-b22a-45e5-8cb7-0acaa9cb6126 2024-03-25T00:27:17.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:27:17.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:27:17.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:27:17.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:27:17.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-03-25T00:27:17.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:27:17.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:27:17.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-25T00:27:17.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:27:17.221 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:27:17.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:27:17.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-03-25T00:27:17.223 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:27:17.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9122384d-7eec-47a0-bb55-fe2408656ce3 2024-03-25T00:27:17.223 INFO:tasks.workunit.client.0.smithi080.stdout:add osd6 9122384d-7eec-47a0-bb55-fe2408656ce3 2024-03-25T00:27:17.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 9122384d-7eec-47a0-bb55-fe2408656ce3' 2024-03-25T00:27:17.224 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:27:17.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDlxABmj2jmDhAAHS2laoNHi7Blnrg9frBSuw== 2024-03-25T00:27:17.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDlxABmj2jmDhAAHS2laoNHi7Blnrg9frBSuw=="}' 2024-03-25T00:27:17.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9122384d-7eec-47a0-bb55-fe2408656ce3 -i td/test-erasure-eio/6/new.json 2024-03-25T00:27:17.923 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:27:17.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-03-25T00:27:17.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlxABmj2jmDhAAHS2laoNHi7Blnrg9frBSuw== --osd-uuid 9122384d-7eec-47a0-bb55-fe2408656ce3 2024-03-25T00:27:17.969 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:17.969+0000 7ff6eea727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:27:18.049 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:18.049+0000 7ff6eea727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:27:18.051 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:18.053+0000 7ff6eea727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:27:18.051 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:18.053+0000 7ff6eea727c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:27:18.051 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:18.053+0000 7ff6eea727c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:27:18.051 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:18.053+0000 7ff6eea727c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:27:18.119 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:18.125+0000 7ff6eea727c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-03-25T00:27:25.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-03-25T00:27:25.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:27:25.607 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd6 key to auth repository 2024-03-25T00:27:25.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-25T00:27:25.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:27:26.164 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.6 2024-03-25T00:27:26.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-25T00:27:26.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:27:26.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:27:26.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:27:26.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:27:26.197 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:26.197+0000 7f40e7bb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:27:26.199 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:26.201+0000 7f40e7bb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:27:26.200 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:26.201+0000 7f40e7bb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:27:26.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-25T00:27:26.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:27:26.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-25T00:27:26.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:27:26.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:27:26.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:26.624 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:27:26.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:27:26.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:26.625 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:26.772 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:26.773+0000 7f40e7bb27c0 -1 Falling back to public interface 2024-03-25T00:27:27.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:28.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:28.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:28.042 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:27:28.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:27:28.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:28.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:28.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:29.424 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:27:29.425+0000 7f40e7bb27c0 -1 osd.6 0 log_to_monitors true 2024-03-25T00:27:29.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:29.465 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:29.465 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:27:29.465 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:27:29.465 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:29.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:30.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:31.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:31.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:31.040 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:27:31.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:27:31.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:31.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:31.809 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:32.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:32.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:32.812 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:27:32.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:27:32.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:32.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:33.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:34.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:34.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:34.384 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:27:34.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:27:34.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:34.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:34.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:35.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:35.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:35.965 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:27:35.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:27:35.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:35.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:36.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:37.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:37.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:37.506 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:27:37.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:27:37.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:37.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:37.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:38.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:38.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:38.933 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:27:38.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:27:38.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:38.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:39.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:40.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:40.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:40.431 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:27:40.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:27:40.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:40.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:40.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:41.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:41.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:41.908 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:27:41.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:27:41.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:41.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:42.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:43.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:43.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:43.431 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:27:43.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:27:43.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:43.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:43.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:44.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:44.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:44.895 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:27:44.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:27:44.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:44.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:45.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:46.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:46.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:46.726 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:27:46.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:27:46.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:46.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:47.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:48.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:48.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:48.590 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:27:48.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:27:48.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:48.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:49.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:50.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:50.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:50.939 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:27:50.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:27:50.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:50.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:51.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:52.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:52.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:52.466 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:27:52.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:27:52.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:52.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:53.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:54.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:54.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:54.004 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:27:54.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:27:54.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:54.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:55.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:27:56.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:27:56.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:27:56.140 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:27:56.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:27:56.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:27:56.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:27:56.706 INFO:tasks.workunit.client.0.smithi080.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/574393841,v1:127.0.0.1:6851/574393841] [v2:127.0.0.1:6852/574393841,v1:127.0.0.1:6853/574393841] exists,up 9122384d-7eec-47a0-bb55-fe2408656ce3 2024-03-25T00:27:56.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:27:56.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:27:56.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:27:56.707 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-03-25T00:27:56.707 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:27:56.707 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:27:56.708 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:27:56.708 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:27:56.708 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:27:56.708 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.0.asok 2024-03-25T00:27:56.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-03-25T00:27:56.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.0.asok log flush 2024-03-25T00:27:56.822 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:27:56.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-03-25T00:27:56.843 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:23:53.876+0000 7f2464f9f7c0 0 load: jerasure load: lrc 2024-03-25T00:27:56.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2024-03-25T00:27:56.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2024-03-25T00:27:56.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:27:56.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-03-25T00:27:56.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-03-25T00:27:56.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-03-25T00:27:56.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-03-25T00:27:56.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-03-25T00:27:56.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-03-25T00:27:57.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-03-25T00:27:57.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-03-25T00:27:59.314 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' already exists 2024-03-25T00:27:59.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:28:00.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-03-25T00:28:00.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:28:00.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:28:00.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:28:00.333 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:28:00.334 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:28:00.334 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:28:00.335 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:28:00.335 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:28:00.335 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:28:00.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:28:00.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:28:00.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:28:00.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:28:00.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:28:00.456 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:28:00.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:28:00.969 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:28:00.969 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:28:00.969 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:28:00.969 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:28:00.969 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:28:00.969 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:28:00.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:28:00.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:00.970 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:28:01.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803824 2024-03-25T00:28:01.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803824 2024-03-25T00:28:01.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824' 2024-03-25T00:28:01.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:01.142 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:28:01.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574892 2024-03-25T00:28:01.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574892 2024-03-25T00:28:01.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-55834574892' 2024-03-25T00:28:01.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:01.300 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:28:01.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345956 2024-03-25T00:28:01.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345956 2024-03-25T00:28:01.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-55834574892 2-85899345956' 2024-03-25T00:28:01.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:01.450 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:28:01.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018912 2024-03-25T00:28:01.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018912 2024-03-25T00:28:01.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-55834574892 2-85899345956 3-128849018912' 2024-03-25T00:28:01.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:01.719 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:28:01.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691868 2024-03-25T00:28:01.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691868 2024-03-25T00:28:01.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-55834574892 2-85899345956 3-128849018912 4-171798691868' 2024-03-25T00:28:01.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:01.872 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:28:02.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299407 2024-03-25T00:28:02.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299407 2024-03-25T00:28:02.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-55834574892 2-85899345956 3-128849018912 4-171798691868 5-223338299407' 2024-03-25T00:28:02.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:02.028 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:28:02.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005064 2024-03-25T00:28:02.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005064 2024-03-25T00:28:02.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-55834574892 2-85899345956 3-128849018912 4-171798691868 5-223338299407 6-261993005064' 2024-03-25T00:28:02.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:02.179 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803824 2024-03-25T00:28:02.179 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:02.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:28:02.180 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803824 2024-03-25T00:28:02.180 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:02.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803824 2024-03-25T00:28:02.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803824' 2024-03-25T00:28:02.182 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803824 2024-03-25T00:28:02.182 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:28:02.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803824 -lt 25769803824 2024-03-25T00:28:02.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:02.682 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574892 2024-03-25T00:28:02.682 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:02.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:28:02.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:02.684 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574892 2024-03-25T00:28:02.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574892 2024-03-25T00:28:02.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574892' 2024-03-25T00:28:02.686 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 55834574892 2024-03-25T00:28:02.686 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:28:03.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574892 -lt 55834574892 2024-03-25T00:28:03.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:03.129 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345956 2024-03-25T00:28:03.129 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:03.131 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:28:03.132 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345956 2024-03-25T00:28:03.132 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:03.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345956 2024-03-25T00:28:03.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345956' 2024-03-25T00:28:03.134 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 85899345956 2024-03-25T00:28:03.134 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:28:03.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345956 -lt 85899345956 2024-03-25T00:28:03.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:03.544 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018912 2024-03-25T00:28:03.544 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:03.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:28:03.547 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018912 2024-03-25T00:28:03.547 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:03.548 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018912 2024-03-25T00:28:03.548 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 128849018912 2024-03-25T00:28:03.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018912' 2024-03-25T00:28:03.549 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:28:04.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018912 -lt 128849018912 2024-03-25T00:28:04.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:04.020 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-171798691868 2024-03-25T00:28:04.020 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:04.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:28:04.021 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-171798691868 2024-03-25T00:28:04.023 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:04.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691868 2024-03-25T00:28:04.023 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 171798691868 2024-03-25T00:28:04.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 171798691868' 2024-03-25T00:28:04.024 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:28:04.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691869 -lt 171798691868 2024-03-25T00:28:04.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:04.467 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:04.467 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-223338299407 2024-03-25T00:28:04.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:28:04.469 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-223338299407 2024-03-25T00:28:04.469 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:04.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299407 2024-03-25T00:28:04.471 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 223338299407 2024-03-25T00:28:04.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 223338299407' 2024-03-25T00:28:04.471 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:28:04.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299408 -lt 223338299407 2024-03-25T00:28:04.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:04.914 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005064 2024-03-25T00:28:04.914 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:04.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:28:04.916 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:04.916 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005064 2024-03-25T00:28:04.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005064 2024-03-25T00:28:04.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005064' 2024-03-25T00:28:04.918 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 261993005064 2024-03-25T00:28:04.918 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:28:05.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005064 -lt 261993005064 2024-03-25T00:28:05.351 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:28:05.352 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:28:05.352 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:28:05.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:28:05.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:28:05.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:28:05.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:28:05.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:28:05.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:28:05.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:28:05.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:28:06.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:28:06.349 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:28:06.350 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:28:06.350 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:28:06.868 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:28:06.868 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:28:06.868 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:28:06.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2024-03-25T00:28:06.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-03-25T00:28:06.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-03-25T00:28:06.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:28:06.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-25T00:28:07.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: local primary_osd=3 2024-03-25T00:28:07.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2024-03-25T00:28:07.340 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:28:07.340 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:28:07.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:28:07.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:28:07.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:28:07.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:28:07.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-03-25T00:28:07.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:28:07.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:28:07.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-03-25T00:28:07.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:28:07.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:28:07.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:28:07.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:28:07.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:28:07.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:28:07.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:28:07.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:28:07.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-03-25T00:28:08.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:375: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:28:08.804 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-03-25T00:28:08.805 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:28:08.805 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:28:09.271 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2147483647 2024-03-25T00:28:09.271 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:28:09.271 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:28:09.271 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:28:09.271 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:28:09.271 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2147483647 5 1 0 6 2024-03-25T00:28:09.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('2147483647' '5' '1' '0' '6') 2024-03-25T00:28:09.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-25T00:28:09.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=5 2024-03-25T00:28:09.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-03-25T00:28:09.272 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/5/type 2024-03-25T00:28:09.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-25T00:28:09.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2024-03-25T00:28:09.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T00:28:09.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=5 2024-03-25T00:28:09.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-25T00:28:09.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-25T00:28:09.274 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T00:28:09.274 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.5 2024-03-25T00:28:09.274 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-03-25T00:28:09.274 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-03-25T00:28:09.274 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:28:09.274 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:09.274 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:09.275 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.5.asok 2024-03-25T00:28:09.275 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81310/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2024-03-25T00:28:09.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T00:28:09.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-25T00:28:09.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-25T00:28:09.392 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.5 2024-03-25T00:28:09.392 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-03-25T00:28:09.392 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-03-25T00:28:09.392 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:28:09.392 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:09.392 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:09.393 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.5.asok 2024-03-25T00:28:09.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-25T00:28:09.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2024-03-25T00:28:09.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:378: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2024-03-25T00:28:09.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:28:09.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:28:09.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T00:28:09.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:28:09.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:28:09.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:28:09.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:28:09.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:28:09.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:28:09.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:28:09.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:28:09.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:28:09.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:28:09.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:28:09.518 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:28:09.518 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:28:09.518 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:28:09.518 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:28:09.518 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:09.518 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:09.520 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:28:09.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:28:09.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:28:09.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:28:09.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:28:09.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:28:09.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:28:09.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:28:09.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:28:09.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:28:09.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:28:09.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:28:09.521 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.3 2024-03-25T00:28:09.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T00:28:09.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:28:09.522 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-03-25T00:28:09.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T00:28:09.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:28:09.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:28:09.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:28:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:09.551+0000 7f76ca81f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:09.553 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:09.555+0000 7f76ca81f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:09.554 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:09.555+0000 7f76ca81f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:09.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T00:28:09.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:28:09.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:28:09.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:28:09.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:28:09.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:09.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:28:09.972 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:28:09.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:09.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:28:10.108 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:10.111+0000 7f76ca81f7c0 -1 Falling back to public interface 2024-03-25T00:28:10.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:11.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:11.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:11.377 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:28:11.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:28:11.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:11.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:28:11.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:12.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:12.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:12.773 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:28:12.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:28:12.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:12.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:28:13.056 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:13.059+0000 7f76ca81f7c0 -1 osd.3 70 log_to_monitors true 2024-03-25T00:28:13.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:14.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:14.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:14.189 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:28:14.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:28:14.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:14.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:28:14.673 INFO:tasks.workunit.client.0.smithi080.stdout:osd.3 up in weight 1 up_from 74 up_thru 74 down_at 71 last_clean_interval [30,70) [v2:127.0.0.1:6826/2022258904,v1:127.0.0.1:6827/2022258904] [v2:127.0.0.1:6828/2022258904,v1:127.0.0.1:6829/2022258904] exists,up 05ec8d40-942e-42b8-9a5a-7d205ccdd13d 2024-03-25T00:28:14.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:28:14.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:28:14.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:28:14.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:381: TEST_ec_object_attr_read_error: wait_for_clean 2024-03-25T00:28:14.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:28:14.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:28:14.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:28:14.675 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:28:14.675 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:28:14.675 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:28:14.675 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:28:14.675 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:28:14.675 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:28:14.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:28:14.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:28:14.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:28:14.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:28:14.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:28:14.773 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:28:15.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:28:15.291 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:28:15.291 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:28:15.292 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:28:15.292 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:28:15.292 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:28:15.292 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:28:15.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:28:15.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:15.292 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:28:15.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803829 2024-03-25T00:28:15.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803829 2024-03-25T00:28:15.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829' 2024-03-25T00:28:15.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:15.445 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:28:15.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574897 2024-03-25T00:28:15.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574897 2024-03-25T00:28:15.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897' 2024-03-25T00:28:15.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:15.593 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:28:15.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345961 2024-03-25T00:28:15.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345961 2024-03-25T00:28:15.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897 2-85899345961' 2024-03-25T00:28:15.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:15.744 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:28:15.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579907 2024-03-25T00:28:15.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579907 2024-03-25T00:28:15.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897 2-85899345961 3-317827579907' 2024-03-25T00:28:15.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:15.900 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:28:16.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691873 2024-03-25T00:28:16.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691873 2024-03-25T00:28:16.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897 2-85899345961 3-317827579907 4-171798691873' 2024-03-25T00:28:16.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:16.052 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:28:16.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299412 2024-03-25T00:28:16.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299412 2024-03-25T00:28:16.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897 2-85899345961 3-317827579907 4-171798691873 5-223338299412' 2024-03-25T00:28:16.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:28:16.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:28:16.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005068 2024-03-25T00:28:16.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005068 2024-03-25T00:28:16.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-55834574897 2-85899345961 3-317827579907 4-171798691873 5-223338299412 6-261993005068' 2024-03-25T00:28:16.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:16.386 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803829 2024-03-25T00:28:16.386 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:16.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:28:16.387 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803829 2024-03-25T00:28:16.387 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:16.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803829 2024-03-25T00:28:16.389 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803829 2024-03-25T00:28:16.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-03-25T00:28:16.389 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:28:16.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803829 -lt 25769803829 2024-03-25T00:28:16.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:16.795 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574897 2024-03-25T00:28:16.795 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:16.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:28:16.797 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574897 2024-03-25T00:28:16.797 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:16.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574897 2024-03-25T00:28:16.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574897' 2024-03-25T00:28:16.799 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 55834574897 2024-03-25T00:28:16.799 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:28:17.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574897 -lt 55834574897 2024-03-25T00:28:17.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:17.207 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345961 2024-03-25T00:28:17.208 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:17.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:28:17.210 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345961 2024-03-25T00:28:17.211 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:17.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345961 2024-03-25T00:28:17.212 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 85899345961 2024-03-25T00:28:17.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345961' 2024-03-25T00:28:17.213 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:28:17.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345961 -lt 85899345961 2024-03-25T00:28:17.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:17.620 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-317827579907 2024-03-25T00:28:17.620 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:17.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:28:17.621 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-317827579907 2024-03-25T00:28:17.621 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:17.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579907 2024-03-25T00:28:17.623 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 317827579907 2024-03-25T00:28:17.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 317827579907' 2024-03-25T00:28:17.623 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:28:18.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579907 2024-03-25T00:28:18.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:18.047 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:18.047 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-171798691873 2024-03-25T00:28:18.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:28:18.049 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-171798691873 2024-03-25T00:28:18.049 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:18.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691873 2024-03-25T00:28:18.051 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 171798691873 2024-03-25T00:28:18.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 171798691873' 2024-03-25T00:28:18.051 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:28:18.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691873 -lt 171798691873 2024-03-25T00:28:18.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:18.480 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-223338299412 2024-03-25T00:28:18.480 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:18.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:28:18.482 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-223338299412 2024-03-25T00:28:18.482 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:18.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299412 2024-03-25T00:28:18.486 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 223338299412 2024-03-25T00:28:18.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 223338299412' 2024-03-25T00:28:18.487 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:28:18.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299413 -lt 223338299412 2024-03-25T00:28:18.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:28:18.916 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005068 2024-03-25T00:28:18.916 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:28:18.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:28:18.918 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005068 2024-03-25T00:28:18.918 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:28:18.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005068 2024-03-25T00:28:18.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005068' 2024-03-25T00:28:18.920 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 261993005068 2024-03-25T00:28:18.920 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:28:19.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005069 -lt 261993005068 2024-03-25T00:28:19.335 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:28:19.335 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:28:19.335 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:28:19.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:28:19.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:28:19.915 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:28:19.915 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:28:19.915 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:28:19.915 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:28:19.915 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:28:19.916 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:28:20.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:28:20.317 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:28:20.317 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:28:20.317 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:28:20.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:28:20.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:28:20.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:28:20.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:383: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-03-25T00:28:20.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:28:20.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:28:20.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-03-25T00:28:20.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:28:20.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:28:20.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-03-25T00:28:20.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:28:20.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:28:20.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2024-03-25T00:28:20.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:28:20.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-03-25T00:28:21.391 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' does not exist 2024-03-25T00:28:21.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-03-25T00:28:22.188 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile myprofile does not exist 2024-03-25T00:28:22.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-03-25T00:28:22.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:28:22.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:28:22.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:28:22.206 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:28:22.206 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:28:22.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:28:22.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:28:22.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:28:22.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:28:22.521 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:28:22.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:28:22.523 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:28:22.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:28:22.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:28:22.525 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:28:22.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:28:22.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:28:22.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:28:22.527 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:28:22.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:28:22.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:28:22.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:28:22.569 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:28:22.569 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:22.570 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:22.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:28:22.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:28:22.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:28:22.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-03-25T00:28:22.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-03-25T00:28:22.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-03-25T00:28:22.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-03-25T00:28:22.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:28:22.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:28:22.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:28:22.572 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:28:22.573 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:28:22.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:28:22.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:28:22.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:28:22.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:28:22.574 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:28:22.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:28:22.576 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:28:22.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:28:22.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:28:22.578 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:28:22.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:28:22.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:28:22.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:28:22.580 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:28:22.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:28:22.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:28:22.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:28:22.587 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:28:22.587 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:22.587 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:22.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:28:22.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:28:22.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:28:22.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-03-25T00:28:22.587 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:28:22.588 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:22.588 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:22.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81310 2024-03-25T00:28:22.589 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:28:22.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:28:22.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:28:22.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-03-25T00:28:22.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-03-25T00:28:22.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-03-25T00:28:22.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:28:22.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:28:22.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:28:22.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-03-25T00:28:22.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-03-25T00:28:23.067 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:28:23.067 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:28:23.067 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:28:23.067 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:28:23.068 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:23.068 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:23.068 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:28:23.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T00:28:23.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:28:23.270 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:28:23.270 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:28:23.270 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:28:23.271 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:28:23.272 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:28:23.272 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:28:23.272 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:28:23.272 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:28:23.272 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:28:23.272 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:23.272 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:23.272 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:28:23.272 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:28:23.273 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get fsid 2024-03-25T00:28:23.381 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:28:23.381 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:28:23.381 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:28:23.381 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:28:23.382 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:28:23.382 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:28:23.382 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:28:23.382 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:28:23.383 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:28:23.383 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:23.383 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:23.383 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:28:23.383 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:28:23.383 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get mon_host 2024-03-25T00:28:23.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-03-25T00:28:23.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-03-25T00:28:23.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:28:23.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:28:23.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:28:23.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-03-25T00:28:23.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:28:24.390 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:28:24.390 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:28:24.390 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:28:24.390 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:28:24.390 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:24.390 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:24.390 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:28:24.391 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:28:24.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:28:24.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-03-25T00:28:24.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T00:28:24.706 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' already exists 2024-03-25T00:28:24.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:28:25.724 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-03-25T00:28:25.724 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:28:25.725 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:28:25.725 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:28:25.725 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:25.725 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:25.725 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:28:25.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-03-25T00:28:25.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-mon.a.asok log flush 2024-03-25T00:28:25.832 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:28:25.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-03-25T00:28:25.848 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:28:23.111+0000 7f8fd5d22c80 0 load: jerasure load: lrc 2024-03-25T00:28:25.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2024-03-25T00:28:25.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2024-03-25T00:28:25.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_recovery_multiple_errors: local objname=myobject 2024-03-25T00:28:25.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_recovery_multiple_errors: setup_osds 9 2024-03-25T00:28:25.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2024-03-25T00:28:25.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-03-25T00:28:25.851 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2024-03-25T00:28:25.851 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2024-03-25T00:28:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:28:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-03-25T00:28:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:28:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:28:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:28:25.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:28:25.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-03-25T00:28:25.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:28:25.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:28:25.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:28:25.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:28:25.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-03-25T00:28:25.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-03-25T00:28:25.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:28:25.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:28:25.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:28:25.856 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:28:25.856 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:28:25.857 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:28:25.857 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:28:25.857 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:25.857 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:25.858 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:28:25.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-03-25T00:28:25.860 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:28:25.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47e7c5f3-46fb-41a5-9087-a0d3427eebd1 2024-03-25T00:28:25.860 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 47e7c5f3-46fb-41a5-9087-a0d3427eebd1 2024-03-25T00:28:25.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 47e7c5f3-46fb-41a5-9087-a0d3427eebd1' 2024-03-25T00:28:25.860 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:28:25.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApxQBm2KTONBAAi16ZD8TNTK6LsHRULRfGtg== 2024-03-25T00:28:25.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApxQBm2KTONBAAi16ZD8TNTK6LsHRULRfGtg=="}' 2024-03-25T00:28:25.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47e7c5f3-46fb-41a5-9087-a0d3427eebd1 -i td/test-erasure-eio/0/new.json 2024-03-25T00:28:26.182 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:28:26.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-03-25T00:28:26.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApxQBm2KTONBAAi16ZD8TNTK6LsHRULRfGtg== --osd-uuid 47e7c5f3-46fb-41a5-9087-a0d3427eebd1 2024-03-25T00:28:26.237 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:26.239+0000 7f165c06c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:26.242 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:26.243+0000 7f165c06c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:26.243 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:26.247+0000 7f165c06c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:26.244 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:26.247+0000 7f165c06c7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:28:26.244 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:26.247+0000 7f165c06c7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:28:26.244 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:26.247+0000 7f165c06c7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:28:26.283 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:26.283+0000 7f165c06c7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-03-25T00:28:31.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-03-25T00:28:31.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:28:31.639 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:28:31.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:28:31.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:28:32.168 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:28:32.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:28:32.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:28:32.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:28:32.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:28:32.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:28:32.202 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:32.203+0000 7f3ab46cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:32.224 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:32.227+0000 7f3ab46cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:32.225 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:32.227+0000 7f3ab46cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:32.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:28:32.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:28:32.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:28:32.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:28:32.584 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:28:32.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:28:32.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:32.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:28:32.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:32.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:32.819 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:32.819+0000 7f3ab46cf7c0 -1 Falling back to public interface 2024-03-25T00:28:33.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:34.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:34.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:34.021 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:28:34.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:28:34.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:34.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:34.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:35.281 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:35.283+0000 7f3ab46cf7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:28:35.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:35.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:35.475 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:28:35.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:28:35.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:35.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:36.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:37.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:37.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:37.013 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:28:37.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:28:37.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:37.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:37.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:38.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:38.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:38.427 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:28:38.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:28:38.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:38.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:38.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:39.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:39.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:39.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:28:39.843 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:28:39.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:39.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:40.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:41.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:41.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:41.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:28:41.362 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:28:41.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:41.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:41.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:42.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:42.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:42.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:28:42.797 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:28:42.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:42.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:43.343 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:44.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:44.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:44.345 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:28:44.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:28:44.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:44.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:44.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:45.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:45.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:45.792 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:28:45.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:28:45.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:45.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:46.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:47.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:47.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:47.239 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:28:47.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:28:47.240 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:47.240 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:47.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:48.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:48.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:48.819 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:28:48.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:28:48.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:48.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:49.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:50.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:50.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:50.943 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:28:50.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:28:50.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:50.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:51.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:52.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:52.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:52.854 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:28:52.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:28:52.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:52.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:53.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:54.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:54.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:54.575 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:28:54.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:28:54.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:54.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:55.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:56.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:56.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:56.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:28:56.338 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:28:56.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:56.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:57.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:28:58.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:28:58.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:28:58.128 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:28:58.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:28:58.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:28:58.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:28:58.575 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1270747195,v1:127.0.0.1:6803/1270747195] [v2:127.0.0.1:6804/1270747195,v1:127.0.0.1:6805/1270747195] exists,up 47e7c5f3-46fb-41a5-9087-a0d3427eebd1 2024-03-25T00:28:58.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:28:58.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:28:58.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:28:58.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:28:58.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-03-25T00:28:58.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:28:58.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:28:58.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T00:28:58.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:28:58.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:28:58.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:28:58.581 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:28:58.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:28:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:28:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:28:58.583 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:28:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ea58237-5721-46b3-a9cb-a003d1caa05d 2024-03-25T00:28:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6ea58237-5721-46b3-a9cb-a003d1caa05d' 2024-03-25T00:28:58.584 INFO:tasks.workunit.client.0.smithi080.stdout:add osd1 6ea58237-5721-46b3-a9cb-a003d1caa05d 2024-03-25T00:28:58.584 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:28:58.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKxQBmyZjAIxAAmXKpB95X9vQVa6cifKG86w== 2024-03-25T00:28:58.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKxQBmyZjAIxAAmXKpB95X9vQVa6cifKG86w=="}' 2024-03-25T00:28:58.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ea58237-5721-46b3-a9cb-a003d1caa05d -i td/test-erasure-eio/1/new.json 2024-03-25T00:28:59.099 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:28:59.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-03-25T00:28:59.116 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKxQBmyZjAIxAAmXKpB95X9vQVa6cifKG86w== --osd-uuid 6ea58237-5721-46b3-a9cb-a003d1caa05d 2024-03-25T00:28:59.151 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:59.150+0000 7f228a97d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:59.152 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:59.154+0000 7f228a97d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:59.154 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:59.154+0000 7f228a97d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:28:59.154 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:59.154+0000 7f228a97d7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:28:59.154 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:59.154+0000 7f228a97d7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:28:59.154 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:59.154+0000 7f228a97d7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:28:59.235 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:28:59.238+0000 7f228a97d7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-03-25T00:29:05.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-03-25T00:29:05.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:29:05.014 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd1 key to auth repository 2024-03-25T00:29:05.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T00:29:05.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:29:05.578 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.1 2024-03-25T00:29:05.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T00:29:05.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:29:05.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:29:05.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:29:05.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:29:05.610 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:05.610+0000 7fd553c597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:29:05.686 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:05.686+0000 7fd553c597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:29:05.687 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:05.690+0000 7fd553c597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:29:05.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T00:29:05.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:29:05.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:29:05.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:29:05.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:29:05.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:05.997 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:29:05.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:29:05.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:05.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:06.254 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:06.254+0000 7fd553c597c0 -1 Falling back to public interface 2024-03-25T00:29:06.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:07.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:07.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:07.407 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:29:07.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:29:07.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:07.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:07.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:08.731 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:08.730+0000 7fd553c597c0 -1 osd.1 0 log_to_monitors true 2024-03-25T00:29:08.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:08.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:08.857 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:29:08.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:29:08.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:08.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:09.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:10.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:10.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:10.402 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:29:10.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:29:10.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:10.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:10.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:11.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:11.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:11.864 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:29:11.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:29:11.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:11.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:12.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:13.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:13.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:13.328 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:29:13.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:29:13.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:13.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:13.968 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:14.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:14.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:14.970 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:29:14.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:29:14.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:14.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:15.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:16.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:16.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:16.516 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:29:16.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:29:16.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:16.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:16.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:17.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:17.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:17.974 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:29:17.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:29:17.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:17.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:18.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:19.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:19.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:19.527 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:29:19.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:29:19.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:19.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:19.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:20.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:20.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:20.971 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:29:20.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:29:20.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:20.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:21.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:22.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:22.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:22.531 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:29:22.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:29:22.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:22.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:23.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:24.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:24.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:24.297 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:29:24.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:29:24.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:24.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:25.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:26.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:26.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:26.014 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:29:26.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:29:26.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:26.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:26.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:27.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:27.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:27.757 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:29:27.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:29:27.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:27.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:28.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:29.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:29.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:29.952 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:29:29.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:29:29.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:29.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:30.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:31.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:31.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:31.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:29:31.948 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:29:31.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:31.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:29:32.452 INFO:tasks.workunit.client.0.smithi080.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4178013417,v1:127.0.0.1:6811/4178013417] [v2:127.0.0.1:6812/4178013417,v1:127.0.0.1:6813/4178013417] exists,up 6ea58237-5721-46b3-a9cb-a003d1caa05d 2024-03-25T00:29:32.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:29:32.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:29:32.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:29:32.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:29:32.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:29:32.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:29:32.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:29:32.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:29:32.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:29:32.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:29:32.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:29:32.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:29:32.459 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:29:32.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae390531-9f93-4a3f-8e07-e7b76c523db9 2024-03-25T00:29:32.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ae390531-9f93-4a3f-8e07-e7b76c523db9' 2024-03-25T00:29:32.460 INFO:tasks.workunit.client.0.smithi080.stdout:add osd2 ae390531-9f93-4a3f-8e07-e7b76c523db9 2024-03-25T00:29:32.460 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:29:32.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBsxQBmNliZHBAA648x2PSm4o4h7e3sF8m7nw== 2024-03-25T00:29:32.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBsxQBmNliZHBAA648x2PSm4o4h7e3sF8m7nw=="}' 2024-03-25T00:29:32.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae390531-9f93-4a3f-8e07-e7b76c523db9 -i td/test-erasure-eio/2/new.json 2024-03-25T00:29:32.968 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:29:32.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-03-25T00:29:32.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsxQBmNliZHBAA648x2PSm4o4h7e3sF8m7nw== --osd-uuid ae390531-9f93-4a3f-8e07-e7b76c523db9 2024-03-25T00:29:33.019 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:33.017+0000 7f666c5c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:29:33.062 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:33.065+0000 7f666c5c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:29:33.064 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:33.065+0000 7f666c5c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:29:33.065 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:33.065+0000 7f666c5c17c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:29:33.065 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:33.065+0000 7f666c5c17c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:29:33.065 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:33.065+0000 7f666c5c17c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:29:33.162 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:33.161+0000 7f666c5c17c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-03-25T00:29:38.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-03-25T00:29:38.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:29:38.918 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd2 key to auth repository 2024-03-25T00:29:38.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T00:29:38.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:29:39.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T00:29:39.466 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:29:39.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:29:39.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:29:39.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:29:39.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:29:39.491 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:39.489+0000 7f2514ced7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:29:39.494 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:39.493+0000 7f2514ced7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:29:39.495 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:39.493+0000 7f2514ced7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:29:39.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T00:29:39.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:29:39.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:29:39.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:29:39.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:29:39.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:39.919 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:29:39.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:29:39.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:39.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:40.057 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:40.057+0000 7f2514ced7c0 -1 Falling back to public interface 2024-03-25T00:29:40.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:41.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:41.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:41.337 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:29:41.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:29:41.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:41.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:41.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:42.596 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:29:42.597+0000 7f2514ced7c0 -1 osd.2 0 log_to_monitors true 2024-03-25T00:29:42.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:42.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:42.763 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:29:42.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:29:42.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:42.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:43.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:44.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:44.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:44.238 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:29:44.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:29:44.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:44.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:44.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:45.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:45.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:45.738 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:29:45.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:29:45.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:45.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:46.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:47.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:47.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:47.214 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:29:47.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:29:47.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:47.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:47.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:48.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:48.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:48.758 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:29:48.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:29:48.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:48.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:49.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:50.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:50.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:50.179 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:29:50.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:29:50.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:50.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:50.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:51.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:51.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:51.701 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:29:51.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:29:51.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:51.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:52.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:53.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:53.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:53.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:29:53.249 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:29:53.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:53.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:53.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:54.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:54.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:54.723 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:29:54.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:29:54.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:54.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:55.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:56.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:56.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:56.127 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:29:56.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:29:56.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:56.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:56.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:57.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:57.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:57.858 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:29:57.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:29:57.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:57.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:29:58.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:29:59.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:29:59.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:29:59.906 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:29:59.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:29:59.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:29:59.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:30:00.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:01.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:01.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:01.927 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:30:01.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:30:01.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:01.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:30:02.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:03.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:03.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:03.689 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:30:03.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:30:03.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:03.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:30:04.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:05.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:05.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:05.571 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:30:05.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:30:05.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:05.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:30:06.291 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1164154650,v1:127.0.0.1:6819/1164154650] [v2:127.0.0.1:6820/1164154650,v1:127.0.0.1:6821/1164154650] exists,up ae390531-9f93-4a3f-8e07-e7b76c523db9 2024-03-25T00:30:06.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:30:06.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:30:06.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:30:06.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:30:06.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-03-25T00:30:06.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:30:06.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:30:06.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T00:30:06.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:30:06.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:30:06.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:30:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:30:06.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:30:06.298 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:30:06.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6c840074-7ca0-4f36-9e9b-721219c20534 2024-03-25T00:30:06.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 6c840074-7ca0-4f36-9e9b-721219c20534' 2024-03-25T00:30:06.298 INFO:tasks.workunit.client.0.smithi080.stdout:add osd3 6c840074-7ca0-4f36-9e9b-721219c20534 2024-03-25T00:30:06.299 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:30:06.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOxQBmnG8zExAAJdWLiptXhNNAdRVfrr+pwg== 2024-03-25T00:30:06.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOxQBmnG8zExAAJdWLiptXhNNAdRVfrr+pwg=="}' 2024-03-25T00:30:06.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6c840074-7ca0-4f36-9e9b-721219c20534 -i td/test-erasure-eio/3/new.json 2024-03-25T00:30:06.890 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:30:06.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-03-25T00:30:06.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOxQBmnG8zExAAJdWLiptXhNNAdRVfrr+pwg== --osd-uuid 6c840074-7ca0-4f36-9e9b-721219c20534 2024-03-25T00:30:06.947 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:06.948+0000 7f94ef3a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:06.950 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:06.952+0000 7f94ef3a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:06.955 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:06.952+0000 7f94ef3a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:06.955 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:06.952+0000 7f94ef3a47c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:30:06.955 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:06.952+0000 7f94ef3a47c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:30:06.955 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:06.952+0000 7f94ef3a47c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:30:07.062 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:07.064+0000 7f94ef3a47c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-03-25T00:30:13.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-03-25T00:30:13.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:30:13.422 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd3 key to auth repository 2024-03-25T00:30:13.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T00:30:13.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:30:14.000 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.3 2024-03-25T00:30:14.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T00:30:14.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:30:14.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:30:14.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:30:14.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:30:14.031 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:14.032+0000 7f0da82ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:14.053 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:14.056+0000 7f0da82ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:14.054 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:14.056+0000 7f0da82ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:14.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T00:30:14.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:30:14.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:30:14.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:30:14.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:30:14.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:14.417 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:30:14.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:30:14.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:14.418 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:14.635 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:14.636+0000 7f0da82ef7c0 -1 Falling back to public interface 2024-03-25T00:30:14.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:15.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:15.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:15.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:30:15.825 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:30:15.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:15.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:16.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:17.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:17.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:17.254 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:30:17.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:30:17.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:17.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:17.298 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:17.300+0000 7f0da82ef7c0 -1 osd.3 0 log_to_monitors true 2024-03-25T00:30:17.800 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:18.801 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:18.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:18.802 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:30:18.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:30:18.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:18.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:19.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:20.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:20.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:20.306 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:30:20.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:30:20.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:20.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:20.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:21.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:21.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:21.745 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:30:21.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:30:21.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:21.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:22.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:23.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:23.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:23.321 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:30:23.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:30:23.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:23.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:23.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:24.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:24.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:24.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:30:24.841 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:30:24.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:24.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:25.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:26.348 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:26.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:26.349 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:30:26.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:30:26.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:26.350 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:26.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:27.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:27.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:27.813 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:30:27.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:30:27.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:27.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:28.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:29.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:29.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:29.284 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:30:29.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:30:29.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:29.285 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:29.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:30.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:30.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:30.858 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:30:30.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:30:30.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:30.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:31.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:32.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:32.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:32.508 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:30:32.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:30:32.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:32.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:33.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:34.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:34.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:34.121 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:30:34.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:30:34.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:34.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:35.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:36.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:36.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:36.078 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:30:36.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:30:36.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:36.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:37.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:38.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:38.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:38.090 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:30:38.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:30:38.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:38.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:38.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:39.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:39.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:39.956 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:30:39.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:30:39.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:39.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:40.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:41.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:41.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:41.751 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:30:41.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:30:41.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:41.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:42.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:43.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:43.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:43.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:30:43.771 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:30:43.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:43.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:30:44.323 INFO:tasks.workunit.client.0.smithi080.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2519282572,v1:127.0.0.1:6827/2519282572] [v2:127.0.0.1:6828/2519282572,v1:127.0.0.1:6829/2519282572] exists,up 6c840074-7ca0-4f36-9e9b-721219c20534 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:30:44.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:30:44.328 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:30:44.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-03-25T00:30:44.330 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:30:44.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0585337c-3019-456d-bd55-dc2c04075585 2024-03-25T00:30:44.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 0585337c-3019-456d-bd55-dc2c04075585' 2024-03-25T00:30:44.330 INFO:tasks.workunit.client.0.smithi080.stdout:add osd4 0585337c-3019-456d-bd55-dc2c04075585 2024-03-25T00:30:44.331 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:30:44.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0xQBm9hclFRAAoQUslBduoVb3/OX31ZHAuw== 2024-03-25T00:30:44.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0xQBm9hclFRAAoQUslBduoVb3/OX31ZHAuw=="}' 2024-03-25T00:30:44.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0585337c-3019-456d-bd55-dc2c04075585 -i td/test-erasure-eio/4/new.json 2024-03-25T00:30:45.271 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:30:45.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-03-25T00:30:45.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0xQBm9hclFRAAoQUslBduoVb3/OX31ZHAuw== --osd-uuid 0585337c-3019-456d-bd55-dc2c04075585 2024-03-25T00:30:45.329 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:45.331+0000 7f1f1d7267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:45.332 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:45.331+0000 7f1f1d7267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:45.333 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:45.335+0000 7f1f1d7267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:45.334 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:45.335+0000 7f1f1d7267c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:30:45.334 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:45.335+0000 7f1f1d7267c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:30:45.334 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:45.335+0000 7f1f1d7267c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:30:45.489 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:45.491+0000 7f1f1d7267c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-03-25T00:30:51.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-03-25T00:30:51.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:30:51.614 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd4 key to auth repository 2024-03-25T00:30:51.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-25T00:30:51.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:30:52.172 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.4 2024-03-25T00:30:52.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-25T00:30:52.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:30:52.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:30:52.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:30:52.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:30:52.198 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:52.199+0000 7feb2120e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:52.223 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:52.223+0000 7feb2120e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:52.224 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:52.227+0000 7feb2120e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:30:52.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-25T00:30:52.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:30:52.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-25T00:30:52.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:30:52.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:30:52.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:52.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:30:52.598 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:30:52.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:30:52.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:52.805 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:52.807+0000 7feb2120e7c0 -1 Falling back to public interface 2024-03-25T00:30:52.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:53.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:53.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:53.998 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:30:53.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:30:53.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:53.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:30:54.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:55.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:55.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:55.408 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:30:55.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:30:55.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:55.409 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:30:55.407+0000 7feb2120e7c0 -1 osd.4 0 log_to_monitors true 2024-03-25T00:30:55.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:30:55.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:56.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:56.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:56.912 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:30:56.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:30:56.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:56.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:30:57.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:30:58.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:30:58.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:30:58.827 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:30:58.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:30:58.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:30:58.828 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:30:59.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:00.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:00.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:00.591 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:31:00.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:31:00.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:00.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:01.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:02.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:02.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:02.038 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:31:02.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:31:02.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:02.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:02.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:03.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:03.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:03.562 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:31:03.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:31:03.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:03.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:04.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:05.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:05.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:05.072 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:31:05.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:31:05.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:05.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:05.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:06.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:06.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:06.580 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:31:06.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:31:06.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:06.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:07.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:08.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:08.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:08.022 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:31:08.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:31:08.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:08.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:08.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:09.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:09.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:09.519 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:31:09.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:31:09.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:09.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:10.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:11.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:11.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:11.498 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:31:11.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:31:11.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:11.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:12.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:13.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:13.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:13.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:31:13.322 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:31:13.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:13.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:14.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:15.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:15.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:15.358 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:31:15.359 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:31:15.359 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:15.359 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:16.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:17.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:17.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:17.344 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:31:17.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:31:17.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:17.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:18.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:19.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:19.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:19.495 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:31:19.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:31:19.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:19.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:20.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:21.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:21.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:21.216 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:31:21.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:31:21.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:21.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:31:21.925 INFO:tasks.workunit.client.0.smithi080.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1349453359,v1:127.0.0.1:6835/1349453359] [v2:127.0.0.1:6836/1349453359,v1:127.0.0.1:6837/1349453359] exists,up 0585337c-3019-456d-bd55-dc2c04075585 2024-03-25T00:31:21.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:31:21.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:31:21.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:31:21.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:31:21.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:31:21.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:31:21.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:31:21.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:31:21.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:31:21.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:31:21.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:31:21.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:31:21.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:31:21.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:31:21.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-03-25T00:31:21.933 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:31:21.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c69d8ab1-3d39-434a-9c69-f0e812079dfc 2024-03-25T00:31:21.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 c69d8ab1-3d39-434a-9c69-f0e812079dfc' 2024-03-25T00:31:21.934 INFO:tasks.workunit.client.0.smithi080.stdout:add osd5 c69d8ab1-3d39-434a-9c69-f0e812079dfc 2024-03-25T00:31:21.934 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:31:21.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZxQBmaz4KORAA8iFhtO+dScuIy+CUQnb9Bg== 2024-03-25T00:31:21.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZxQBmaz4KORAA8iFhtO+dScuIy+CUQnb9Bg=="}' 2024-03-25T00:31:21.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c69d8ab1-3d39-434a-9c69-f0e812079dfc -i td/test-erasure-eio/5/new.json 2024-03-25T00:31:22.715 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:31:22.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-03-25T00:31:22.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZxQBmaz4KORAA8iFhtO+dScuIy+CUQnb9Bg== --osd-uuid c69d8ab1-3d39-434a-9c69-f0e812079dfc 2024-03-25T00:31:22.773 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:22.774+0000 7f70946dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:31:22.851 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:22.850+0000 7f70946dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:31:22.853 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:22.854+0000 7f70946dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:31:22.853 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:22.854+0000 7f70946dd7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:31:22.853 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:22.854+0000 7f70946dd7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:31:22.854 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:22.854+0000 7f70946dd7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:31:22.993 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:22.994+0000 7f70946dd7c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-03-25T00:31:29.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-03-25T00:31:29.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:31:29.545 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd5 key to auth repository 2024-03-25T00:31:29.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-25T00:31:29.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:31:30.090 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.5 2024-03-25T00:31:30.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-25T00:31:30.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:31:30.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:31:30.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:31:30.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:31:30.121 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:30.122+0000 7fdcad9a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:31:30.123 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:30.122+0000 7fdcad9a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:31:30.125 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:30.126+0000 7fdcad9a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:31:30.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-25T00:31:30.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:31:30.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T00:31:30.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:31:30.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:31:30.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:30.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:31:30.520 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:31:30.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:30.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:30.674 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:30.674+0000 7fdcad9a87c0 -1 Falling back to public interface 2024-03-25T00:31:30.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:31.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:31.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:31.962 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:31:31.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:31:31.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:31.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:32.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:33.263 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:33.262+0000 7fdcad9a87c0 -1 osd.5 0 log_to_monitors true 2024-03-25T00:31:33.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:33.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:33.364 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:31:33.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:31:33.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:33.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:33.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:34.921 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:34.921 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:34.922 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:31:34.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:31:34.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:34.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:35.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:36.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:36.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:36.731 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:31:36.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:31:36.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:36.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:37.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:38.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:38.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:38.149 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:31:38.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:31:38.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:38.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:38.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:39.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:39.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:39.599 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:31:39.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:31:39.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:39.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:40.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:41.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:41.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:41.227 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:31:41.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:31:41.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:41.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:41.801 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:42.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:42.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:42.804 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:31:42.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:31:42.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:42.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:43.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:44.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:44.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:44.389 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:31:44.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:31:44.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:44.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:44.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:45.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:45.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:45.826 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:31:45.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:31:45.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:45.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:46.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:47.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:47.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:47.311 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:31:47.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:31:47.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:47.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:48.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:49.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:49.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:49.455 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:31:49.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:31:49.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:49.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:50.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:51.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:51.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:51.020 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:31:51.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:31:51.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:51.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:52.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:53.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:53.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:53.114 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:31:53.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:31:53.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:53.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:53.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:54.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:54.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:54.920 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:31:54.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:31:54.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:54.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:55.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:56.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:56.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:56.972 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:31:56.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:31:56.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:56.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:57.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:31:58.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:31:58.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:31:58.380 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:31:58.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:31:58.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:31:58.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:31:58.987 INFO:tasks.workunit.client.0.smithi080.stdout:osd.5 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2894364035,v1:127.0.0.1:6843/2894364035] [v2:127.0.0.1:6844/2894364035,v1:127.0.0.1:6845/2894364035] exists,up c69d8ab1-3d39-434a-9c69-f0e812079dfc 2024-03-25T00:31:58.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:31:58.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:31:58.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:31:58.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:31:58.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-03-25T00:31:58.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:31:58.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:31:58.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-25T00:31:58.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:31:58.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-03-25T00:31:58.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:31:58.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:31:58.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:31:58.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:31:58.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-03-25T00:31:58.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-03-25T00:31:58.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:31:58.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:31:58.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:31:58.992 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:31:58.992 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:31:58.993 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:31:58.993 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:31:58.993 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:31:58.993 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:31:58.993 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:31:58.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:31:58.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:31:58.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:31:58.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:31:58.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:31:58.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:31:58.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:31:58.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:31:58.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:31:58.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:31:58.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:31:58.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:31:58.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-03-25T00:31:58.995 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:31:58.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ac8eb44-6de9-4465-9322-42b5fc5ab87b 2024-03-25T00:31:58.995 INFO:tasks.workunit.client.0.smithi080.stdout:add osd6 7ac8eb44-6de9-4465-9322-42b5fc5ab87b 2024-03-25T00:31:58.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 7ac8eb44-6de9-4465-9322-42b5fc5ab87b' 2024-03-25T00:31:58.996 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:31:59.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/xQBmCR0MARAAt5LYrfYQf2KSE0y9tHFgfQ== 2024-03-25T00:31:59.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/xQBmCR0MARAAt5LYrfYQf2KSE0y9tHFgfQ=="}' 2024-03-25T00:31:59.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ac8eb44-6de9-4465-9322-42b5fc5ab87b -i td/test-erasure-eio/6/new.json 2024-03-25T00:31:59.834 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:31:59.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-03-25T00:31:59.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/xQBmCR0MARAAt5LYrfYQf2KSE0y9tHFgfQ== --osd-uuid 7ac8eb44-6de9-4465-9322-42b5fc5ab87b 2024-03-25T00:31:59.896 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:59.893+0000 7f636093e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:31:59.898 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:59.897+0000 7f636093e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:31:59.899 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:59.897+0000 7f636093e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:31:59.899 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:59.897+0000 7f636093e7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:31:59.899 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:59.897+0000 7f636093e7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:31:59.900 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:59.897+0000 7f636093e7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:31:59.975 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:31:59.973+0000 7f636093e7c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-03-25T00:32:07.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-03-25T00:32:07.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:32:07.062 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd6 key to auth repository 2024-03-25T00:32:07.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-25T00:32:07.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:32:07.619 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.6 2024-03-25T00:32:07.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-25T00:32:07.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:32:07.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:32:07.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:32:07.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:32:07.650 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:07.649+0000 7f96c604d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:32:07.653 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:07.653+0000 7f96c604d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:32:07.654 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:07.653+0000 7f96c604d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:32:08.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-25T00:32:08.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:32:08.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-25T00:32:08.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:32:08.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:32:08.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:08.049 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:32:08.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:32:08.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:08.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:08.235 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:08.233+0000 7f96c604d7c0 -1 Falling back to public interface 2024-03-25T00:32:08.465 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:09.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:09.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:09.468 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:32:09.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:32:09.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:09.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:09.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:10.588 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:10.585+0000 7f96c604d7c0 -1 osd.6 0 log_to_monitors true 2024-03-25T00:32:10.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:10.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:10.932 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:32:10.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:32:10.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:10.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:11.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:12.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:12.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:12.579 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:32:12.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:32:12.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:12.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:13.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:14.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:14.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:14.356 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:32:14.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:32:14.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:14.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:14.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:15.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:15.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:15.786 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:32:15.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:32:15.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:15.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:16.504 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:17.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:17.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:17.506 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:32:17.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:32:17.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:17.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:18.058 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:19.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:19.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:19.060 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:32:19.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:32:19.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:19.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:19.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:20.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:20.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:20.607 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:32:20.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:32:20.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:20.608 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:21.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:22.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:22.021 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:32:22.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:32:22.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:22.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:22.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:23.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:23.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:23.574 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:32:23.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:32:23.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:23.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:24.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:25.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:25.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:25.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:32:25.316 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:32:25.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:25.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:25.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:26.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:26.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:26.938 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:32:26.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:32:26.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:26.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:28.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:29.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:29.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:29.061 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:32:29.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:32:29.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:29.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:30.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:31.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:31.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:31.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:32:31.133 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:32:31.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:31.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:32.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:33.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:33.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:33.053 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:32:33.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:32:33.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:33.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:33.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:34.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:34.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:34.678 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:32:34.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:32:34.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:34.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:35.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:36.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:36.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:36.281 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:32:36.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:32:36.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:36.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:32:36.706 INFO:tasks.workunit.client.0.smithi080.stdout:osd.6 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1754372380,v1:127.0.0.1:6851/1754372380] [v2:127.0.0.1:6852/1754372380,v1:127.0.0.1:6853/1754372380] exists,up 7ac8eb44-6de9-4465-9322-42b5fc5ab87b 2024-03-25T00:32:36.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:32:36.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:32:36.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:32:36.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/7 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2024-03-25T00:32:36.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2024-03-25T00:32:36.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:32:36.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:32:36.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:32:36.711 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:32:36.711 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:32:36.711 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:32:36.711 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:32:36.711 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:32:36.711 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:32:36.711 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:32:36.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/7 2024-03-25T00:32:36.713 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:32:36.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8291efd5-71df-4d98-8cde-f37843eb8231 2024-03-25T00:32:36.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 8291efd5-71df-4d98-8cde-f37843eb8231' 2024-03-25T00:32:36.714 INFO:tasks.workunit.client.0.smithi080.stdout:add osd7 8291efd5-71df-4d98-8cde-f37843eb8231 2024-03-25T00:32:36.714 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:32:36.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAkxgBmefTVKxAAlUVsTa3oLhezUZSOhA+hMA== 2024-03-25T00:32:36.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAkxgBmefTVKxAAlUVsTa3oLhezUZSOhA+hMA=="}' 2024-03-25T00:32:36.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8291efd5-71df-4d98-8cde-f37843eb8231 -i td/test-erasure-eio/7/new.json 2024-03-25T00:32:37.346 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:32:37.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/7/new.json 2024-03-25T00:32:37.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkxgBmefTVKxAAlUVsTa3oLhezUZSOhA+hMA== --osd-uuid 8291efd5-71df-4d98-8cde-f37843eb8231 2024-03-25T00:32:37.392 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:37.392+0000 7f0a5809e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:32:37.406 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:37.408+0000 7f0a5809e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:32:37.407 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:37.408+0000 7f0a5809e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:32:37.408 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:37.408+0000 7f0a5809e7c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-03-25T00:32:37.408 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:37.408+0000 7f0a5809e7c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-03-25T00:32:37.408 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:37.408+0000 7f0a5809e7c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-03-25T00:32:37.446 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:37.448+0000 7f0a5809e7c0 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2024-03-25T00:32:43.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2024-03-25T00:32:43.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:32:43.800 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd7 key to auth repository 2024-03-25T00:32:43.800 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-25T00:32:43.800 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:32:44.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-25T00:32:44.339 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.7 2024-03-25T00:32:44.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:32:44.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:32:44.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:32:44.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:32:44.365 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:44.368+0000 7ff3be7cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:32:44.393 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:44.392+0000 7ff3be7cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:32:44.394 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:44.396+0000 7ff3be7cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:32:44.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-25T00:32:44.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:32:44.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-25T00:32:44.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:32:44.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:32:44.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:44.736 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:32:44.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:32:44.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:44.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:32:44.967 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:44.968+0000 7ff3be7cb7c0 -1 Falling back to public interface 2024-03-25T00:32:45.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:46.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:46.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:46.205 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:32:46.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:32:46.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:46.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:32:46.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:47.571 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:32:47.552+0000 7ff3be7cb7c0 -1 osd.7 0 log_to_monitors true 2024-03-25T00:32:47.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:47.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:47.624 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:32:47.625 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:32:47.625 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:47.625 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:32:48.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:49.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:49.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:49.239 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:32:49.240 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:32:49.240 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:49.240 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:32:49.848 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:50.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:50.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:50.850 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:32:50.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:32:50.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:50.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:32:51.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:52.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:52.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:52.389 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:32:52.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:32:52.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:52.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:32:52.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:53.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:53.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:53.808 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:32:53.809 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:32:53.809 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:53.809 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:32:54.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:55.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:55.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:55.423 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:32:55.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:32:55.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:55.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:32:55.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:56.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:56.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:56.932 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:32:56.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:32:56.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:56.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:32:57.413 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:58.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:58.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:58.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:32:58.415 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:32:58.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:58.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:32:58.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:32:59.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:32:59.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:32:59.882 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:32:59.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:32:59.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:32:59.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:33:00.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:01.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:01.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:01.464 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:33:01.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:33:01.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:01.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:33:02.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:03.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:03.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:03.093 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:33:03.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:33:03.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:03.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:33:04.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:05.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:05.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:05.049 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:33:05.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:33:05.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:05.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:33:05.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:06.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:06.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:06.591 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:33:06.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:33:06.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:06.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:33:07.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:08.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:08.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:08.388 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:33:08.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:33:08.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:08.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:33:09.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:10.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:10.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:10.291 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:33:10.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:33:10.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:10.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:33:11.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:12.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:12.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:12.045 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:33:12.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:33:12.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:12.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:33:12.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:13.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:13.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:13.912 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:33:13.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:33:13.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:13.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-25T00:33:14.336 INFO:tasks.workunit.client.0.smithi080.stdout:osd.7 up in weight 1 up_from 69 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3946946083,v1:127.0.0.1:6859/3946946083] [v2:127.0.0.1:6860/3946946083,v1:127.0.0.1:6861/3946946083] exists,up 8291efd5-71df-4d98-8cde-f37843eb8231 2024-03-25T00:33:14.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:33:14.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:33:14.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:33:14.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:33:14.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/8 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:33:14.340 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:33:14.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:33:14.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:33:14.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:33:14.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:33:14.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:33:14.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:33:14.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/8 2024-03-25T00:33:14.342 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:33:14.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63cb49e8-f2aa-4706-b650-2b89f962a157 2024-03-25T00:33:14.343 INFO:tasks.workunit.client.0.smithi080.stdout:add osd8 63cb49e8-f2aa-4706-b650-2b89f962a157 2024-03-25T00:33:14.343 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 63cb49e8-f2aa-4706-b650-2b89f962a157' 2024-03-25T00:33:14.343 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:33:14.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKxgBmO7nEFRAAU1KX+inRHU2ekGx2oIvrYw== 2024-03-25T00:33:14.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKxgBmO7nEFRAAU1KX+inRHU2ekGx2oIvrYw=="}' 2024-03-25T00:33:14.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63cb49e8-f2aa-4706-b650-2b89f962a157 -i td/test-erasure-eio/8/new.json 2024-03-25T00:33:15.349 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:33:15.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/8/new.json 2024-03-25T00:33:15.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKxgBmO7nEFRAAU1KX+inRHU2ekGx2oIvrYw== --osd-uuid 63cb49e8-f2aa-4706-b650-2b89f962a157 2024-03-25T00:33:15.406 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:15.407+0000 7f44628367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:33:15.408 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:15.407+0000 7f44628367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:33:15.409 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:15.407+0000 7f44628367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:33:15.410 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:15.411+0000 7f44628367c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-03-25T00:33:15.410 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:15.411+0000 7f44628367c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-03-25T00:33:15.410 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:15.411+0000 7f44628367c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-03-25T00:33:15.487 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:15.487+0000 7f44628367c0 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2024-03-25T00:33:22.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2024-03-25T00:33:22.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:33:22.870 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd8 key to auth repository 2024-03-25T00:33:22.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-25T00:33:22.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:33:23.413 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.8 2024-03-25T00:33:23.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-25T00:33:23.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:33:23.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:33:23.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:33:23.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:33:23.445 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:23.443+0000 7f29a641d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:33:23.447 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:23.447+0000 7f29a641d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:33:23.448 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:23.447+0000 7f29a641d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:33:23.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-25T00:33:23.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:33:23.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-25T00:33:23.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:33:23.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:33:23.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:23.899 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:33:23.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:33:23.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:23.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:24.002 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:24.003+0000 7f29a641d7c0 -1 Falling back to public interface 2024-03-25T00:33:24.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:25.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:25.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:25.309 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:33:25.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:33:25.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:25.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:25.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:26.654 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:33:26.655+0000 7f29a641d7c0 -1 osd.8 0 log_to_monitors true 2024-03-25T00:33:26.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:26.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:26.765 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:33:26.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:33:26.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:26.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:27.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:28.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:28.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:28.369 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:33:28.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:33:28.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:28.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:29.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:30.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:30.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:30.142 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:33:30.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:33:30.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:30.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:30.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:31.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:31.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:31.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:33:31.976 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:33:31.977 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:31.977 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:32.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:33.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:33.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:33.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:33:33.462 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:33:33.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:33.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:33.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:34.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:34.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:34.939 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:33:34.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:33:34.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:34.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:35.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:36.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:36.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:36.512 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:33:36.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:33:36.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:36.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:36.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:37.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:37.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:37.945 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:33:37.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:33:37.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:37.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:38.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:39.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:39.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:39.502 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:33:39.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:33:39.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:39.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:40.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:41.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:41.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:41.004 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:33:41.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:33:41.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:41.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:41.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:42.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:42.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:42.993 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:33:42.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:33:42.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:42.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:43.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:44.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:44.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:44.908 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:33:44.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:33:44.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:44.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:45.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:46.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:46.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:46.785 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:33:46.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:33:46.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:46.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:47.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:48.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:48.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:48.656 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:33:48.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:33:48.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:48.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:49.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:50.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:50.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:50.469 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:33:50.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:33:50.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:50.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:51.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:33:52.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:33:52.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:33:52.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:33:52.424 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:33:52.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:33:52.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-25T00:33:52.922 INFO:tasks.workunit.client.0.smithi080.stdout:osd.8 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/2308462490,v1:127.0.0.1:6867/2308462490] [v2:127.0.0.1:6868/2308462490,v1:127.0.0.1:6869/2308462490] exists,up 63cb49e8-f2aa-4706-b650-2b89f962a157 2024-03-25T00:33:52.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:33:52.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:33:52.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:33:52.924 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-03-25T00:33:52.924 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:33:52.924 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:33:52.925 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:33:52.925 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:33:52.925 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:33:52.925 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.0.asok 2024-03-25T00:33:52.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-03-25T00:33:52.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.0.asok log flush 2024-03-25T00:33:53.038 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:33:53.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-03-25T00:33:53.060 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:28:32.823+0000 7f3ab46cf7c0 0 load: jerasure load: lrc 2024-03-25T00:33:53.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:423: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2024-03-25T00:33:53.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2024-03-25T00:33:53.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:33:53.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-03-25T00:33:53.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2024-03-25T00:33:53.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-03-25T00:33:53.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2024-03-25T00:33:53.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-03-25T00:33:53.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2024-03-25T00:33:54.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-03-25T00:33:54.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-03-25T00:33:55.831 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' already exists 2024-03-25T00:33:55.848 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:33:56.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-03-25T00:33:56.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:33:56.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:33:56.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:33:56.851 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:33:56.852 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:33:56.852 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:33:56.852 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:33:56.853 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:33:56.853 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:33:56.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:33:56.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:33:56.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:33:56.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:33:56.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:33:56.980 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:33:57.403 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:33:57.403 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:33:57.403 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:33:57.403 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:33:57.403 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:33:57.403 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:33:57.403 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:33:57.403 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:33:57.403 INFO:tasks.workunit.client.0.smithi080.stderr:8' 2024-03-25T00:33:57.403 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:33:57.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:33:57.404 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:33:57.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803845 2024-03-25T00:33:57.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803845 2024-03-25T00:33:57.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845' 2024-03-25T00:33:57.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:33:57.655 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:33:57.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574908 2024-03-25T00:33:57.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574908 2024-03-25T00:33:57.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908' 2024-03-25T00:33:57.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:33:57.827 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:33:57.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378675 2024-03-25T00:33:57.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378675 2024-03-25T00:33:57.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-81604378675' 2024-03-25T00:33:57.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:33:57.981 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:33:58.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117036 2024-03-25T00:33:58.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117036 2024-03-25T00:33:58.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-81604378675 3-115964117036' 2024-03-25T00:33:58.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:33:58.127 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:33:58.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757284 2024-03-25T00:33:58.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757284 2024-03-25T00:33:58.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-81604378675 3-115964117036 4-163208757284' 2024-03-25T00:33:58.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:33:58.267 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:33:58.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397534 2024-03-25T00:33:58.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397534 2024-03-25T00:33:58.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-81604378675 3-115964117036 4-163208757284 5-210453397534' 2024-03-25T00:33:58.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:33:58.423 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:33:58.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037785 2024-03-25T00:33:58.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037785 2024-03-25T00:33:58.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-81604378675 3-115964117036 4-163208757284 5-210453397534 6-257698037785' 2024-03-25T00:33:58.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:33:58.572 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T00:33:58.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743438 2024-03-25T00:33:58.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743438 2024-03-25T00:33:58.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-81604378675 3-115964117036 4-163208757284 5-210453397534 6-257698037785 7-296352743438' 2024-03-25T00:33:58.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:33:58.721 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T00:33:58.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383687 2024-03-25T00:33:58.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383687 2024-03-25T00:33:58.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-81604378675 3-115964117036 4-163208757284 5-210453397534 6-257698037785 7-296352743438 8-343597383687' 2024-03-25T00:33:58.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:33:58.873 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803845 2024-03-25T00:33:58.874 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:33:58.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:33:58.875 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803845 2024-03-25T00:33:58.875 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:33:58.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803845 2024-03-25T00:33:58.876 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803845 2024-03-25T00:33:58.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803845' 2024-03-25T00:33:58.877 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:33:59.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803846 -lt 25769803845 2024-03-25T00:33:59.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:33:59.298 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574908 2024-03-25T00:33:59.298 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:33:59.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:33:59.300 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574908 2024-03-25T00:33:59.300 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:33:59.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574908 2024-03-25T00:33:59.302 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 55834574908 2024-03-25T00:33:59.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574908' 2024-03-25T00:33:59.303 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:33:59.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574908 -lt 55834574908 2024-03-25T00:33:59.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:33:59.755 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:33:59.755 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378675 2024-03-25T00:33:59.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:33:59.757 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378675 2024-03-25T00:33:59.757 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:33:59.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378675 2024-03-25T00:33:59.758 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 81604378675 2024-03-25T00:33:59.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378675' 2024-03-25T00:33:59.759 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:34:00.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378675 -lt 81604378675 2024-03-25T00:34:00.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:00.167 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117036 2024-03-25T00:34:00.168 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:00.168 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:34:00.169 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117036 2024-03-25T00:34:00.169 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:00.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117036 2024-03-25T00:34:00.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117036' 2024-03-25T00:34:00.171 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 115964117036 2024-03-25T00:34:00.171 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:34:00.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117037 -lt 115964117036 2024-03-25T00:34:00.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:00.593 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757284 2024-03-25T00:34:00.593 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:00.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:34:00.595 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757284 2024-03-25T00:34:00.595 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:00.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757284 2024-03-25T00:34:00.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757284' 2024-03-25T00:34:00.597 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 163208757284 2024-03-25T00:34:00.597 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:34:01.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757284 -lt 163208757284 2024-03-25T00:34:01.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:01.031 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-210453397534 2024-03-25T00:34:01.031 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:01.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:34:01.034 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-210453397534 2024-03-25T00:34:01.034 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:01.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397534 2024-03-25T00:34:01.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 210453397534' 2024-03-25T00:34:01.036 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 210453397534 2024-03-25T00:34:01.036 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:34:01.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397535 -lt 210453397534 2024-03-25T00:34:01.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:01.444 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-257698037785 2024-03-25T00:34:01.444 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:01.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:34:01.446 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-257698037785 2024-03-25T00:34:01.446 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:01.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037785 2024-03-25T00:34:01.447 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 257698037785 2024-03-25T00:34:01.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 257698037785' 2024-03-25T00:34:01.448 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:34:01.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037785 -lt 257698037785 2024-03-25T00:34:01.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:01.890 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-296352743438 2024-03-25T00:34:01.890 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:01.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-25T00:34:01.892 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-296352743438 2024-03-25T00:34:01.892 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:01.893 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743438 2024-03-25T00:34:01.893 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 296352743438' 2024-03-25T00:34:01.893 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.7 seq 296352743438 2024-03-25T00:34:01.894 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-25T00:34:02.318 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743439 -lt 296352743438 2024-03-25T00:34:02.318 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:02.319 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-343597383687 2024-03-25T00:34:02.319 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:02.320 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T00:34:02.321 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:02.321 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-343597383687 2024-03-25T00:34:02.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383687 2024-03-25T00:34:02.322 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.8 seq 343597383687 2024-03-25T00:34:02.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 343597383687' 2024-03-25T00:34:02.322 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T00:34:02.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383687 -lt 343597383687 2024-03-25T00:34:02.728 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:34:02.728 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:34:02.728 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:34:03.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:34:03.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:34:03.311 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:34:03.312 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:34:03.312 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:34:03.312 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:34:03.312 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:34:03.312 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:34:03.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:34:03.732 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:34:03.732 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:34:03.733 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:34:04.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:34:04.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:34:04.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:34:04.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:34:04.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-03-25T00:34:04.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:427: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-03-25T00:34:04.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-03-25T00:34:04.502 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:34:04.503 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-03-25T00:34:04.503 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:34:04.503 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:34:04.918 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:34:04.918 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:34:04.918 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:34:04.918 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:34:04.918 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:34:04.918 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:34:04.918 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:34:04.918 INFO:tasks.workunit.client.0.smithi080.stderr:7' 2024-03-25T00:34:04.918 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-03-25T00:34:04.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-03-25T00:34:04.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-25T00:34:04.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-03-25T00:34:04.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-03-25T00:34:04.920 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-03-25T00:34:04.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-25T00:34:04.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-03-25T00:34:04.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T00:34:04.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-03-25T00:34:04.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-25T00:34:04.920 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-25T00:34:04.921 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T00:34:04.921 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-03-25T00:34:04.921 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-03-25T00:34:04.921 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-03-25T00:34:04.922 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:34:04.922 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:04.922 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:04.922 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.3.asok 2024-03-25T00:34:04.922 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81310/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-03-25T00:34:05.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T00:34:05.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-25T00:34:05.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-25T00:34:05.031 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-03-25T00:34:05.031 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-03-25T00:34:05.031 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-03-25T00:34:05.031 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:34:05.031 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:05.031 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:05.031 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.3.asok 2024-03-25T00:34:05.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-25T00:34:05.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-03-25T00:34:05.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:430: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2024-03-25T00:34:05.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-03-25T00:34:05.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-25T00:34:05.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-25T00:34:05.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-25T00:34:05.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-03-25T00:34:05.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-25T00:34:05.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-03-25T00:34:05.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-25T00:34:05.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-03-25T00:34:05.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-25T00:34:05.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=3 2024-03-25T00:34:05.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-25T00:34:05.156 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-03-25T00:34:05.156 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:34:05.156 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-03-25T00:34:05.156 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:34:05.156 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:34:05.546 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:34:05.546 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr:7' 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-03-25T00:34:05.547 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/0/type 2024-03-25T00:34:05.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-25T00:34:05.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-25T00:34:05.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T00:34:05.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-25T00:34:05.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-25T00:34:05.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-25T00:34:05.549 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T00:34:05.549 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-25T00:34:05.549 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:34:05.549 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:34:05.550 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:34:05.550 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:05.550 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:05.550 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.0.asok 2024-03-25T00:34:05.551 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81310/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-25T00:34:05.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T00:34:05.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-25T00:34:05.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-25T00:34:05.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-25T00:34:05.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:34:05.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:34:05.674 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:34:05.675 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:05.675 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:05.675 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.0.asok 2024-03-25T00:34:05.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-25T00:34:05.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2024-03-25T00:34:05.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2024-03-25T00:34:05.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-03-25T00:34:05.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-25T00:34:05.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-25T00:34:05.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-25T00:34:05.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-03-25T00:34:05.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-25T00:34:05.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-03-25T00:34:05.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-25T00:34:05.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-03-25T00:34:05.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-25T00:34:05.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=4 2024-03-25T00:34:05.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-25T00:34:05.797 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-03-25T00:34:05.797 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:34:05.797 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-03-25T00:34:05.797 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:34:05.797 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:34:06.214 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:34:06.214 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr:7' 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=6 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-03-25T00:34:06.215 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/6/type 2024-03-25T00:34:06.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-25T00:34:06.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2024-03-25T00:34:06.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T00:34:06.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=6 2024-03-25T00:34:06.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-25T00:34:06.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-25T00:34:06.217 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T00:34:06.217 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.6 2024-03-25T00:34:06.217 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-03-25T00:34:06.217 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-03-25T00:34:06.218 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:34:06.218 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:06.218 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:06.218 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.6.asok 2024-03-25T00:34:06.219 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81310/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2024-03-25T00:34:06.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T00:34:06.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-25T00:34:06.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-25T00:34:06.342 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.6 2024-03-25T00:34:06.343 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-03-25T00:34:06.343 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-03-25T00:34:06.343 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:34:06.343 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:06.343 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:06.344 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.6.asok 2024-03-25T00:34:06.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-25T00:34:06.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2024-03-25T00:34:06.453 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2024-03-25T00:34:06.454 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:34:06.454 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-03-25T00:34:06.454 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:34:06.454 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr:7' 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: local -a initial_osds 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:434: TEST_ec_recovery_multiple_errors: local last_osd=7 2024-03-25T00:34:06.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:436: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2024-03-25T00:34:06.843 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:34:06.843 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:34:06.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:34:06.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:34:06.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:34:07.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:34:07.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:437: TEST_ec_recovery_multiple_errors: ceph osd down 7 2024-03-25T00:34:07.887 INFO:tasks.workunit.client.0.smithi080.stderr:osd.7 is already down. 2024-03-25T00:34:07.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: ceph osd out 7 2024-03-25T00:34:08.635 INFO:tasks.workunit.client.0.smithi080.stderr:osd.7 is already out. 2024-03-25T00:34:08.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: wait_for_clean 2024-03-25T00:34:08.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:34:08.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:34:08.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:34:08.655 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:34:08.655 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:34:08.655 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:34:08.655 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:34:08.655 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:34:08.655 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:34:08.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:34:08.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:34:08.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:34:08.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:34:08.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:34:08.756 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:34:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:34:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:34:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:34:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:34:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:34:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:34:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:6 2024-03-25T00:34:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:7 2024-03-25T00:34:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:8' 2024-03-25T00:34:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:34:09.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:34:09.551 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:34:09.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803850 2024-03-25T00:34:09.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803850 2024-03-25T00:34:09.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850' 2024-03-25T00:34:09.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:34:09.900 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:34:10.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574912 2024-03-25T00:34:10.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574912 2024-03-25T00:34:10.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-55834574912' 2024-03-25T00:34:10.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:34:10.115 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:34:10.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378679 2024-03-25T00:34:10.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378679 2024-03-25T00:34:10.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-55834574912 2-81604378679' 2024-03-25T00:34:10.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:34:10.275 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:34:10.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117041 2024-03-25T00:34:10.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117041 2024-03-25T00:34:10.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-55834574912 2-81604378679 3-115964117041' 2024-03-25T00:34:10.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:34:10.497 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:34:10.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757288 2024-03-25T00:34:10.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757288 2024-03-25T00:34:10.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-55834574912 2-81604378679 3-115964117041 4-163208757288' 2024-03-25T00:34:10.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:34:10.790 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:34:11.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397539 2024-03-25T00:34:11.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397539 2024-03-25T00:34:11.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-55834574912 2-81604378679 3-115964117041 4-163208757288 5-210453397539' 2024-03-25T00:34:11.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:34:11.016 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:34:11.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037790 2024-03-25T00:34:11.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037790 2024-03-25T00:34:11.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-55834574912 2-81604378679 3-115964117041 4-163208757288 5-210453397539 6-257698037790' 2024-03-25T00:34:11.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:34:11.179 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-25T00:34:11.289 INFO:tasks.workunit.client.0.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-03-25T00:34:11.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-03-25T00:34:11.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-03-25T00:34:11.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-03-25T00:34:11.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:34:11.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-25T00:34:11.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383691 2024-03-25T00:34:11.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383691 2024-03-25T00:34:11.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-55834574912 2-81604378679 3-115964117041 4-163208757288 5-210453397539 6-257698037790 8-343597383691' 2024-03-25T00:34:11.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:11.446 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803850 2024-03-25T00:34:11.446 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:11.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:34:11.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803850 2024-03-25T00:34:11.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:11.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803850 2024-03-25T00:34:11.448 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803850 2024-03-25T00:34:11.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803850' 2024-03-25T00:34:11.449 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:34:11.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803850 -lt 25769803850 2024-03-25T00:34:11.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:11.862 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574912 2024-03-25T00:34:11.862 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:11.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:34:11.864 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574912 2024-03-25T00:34:11.864 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:11.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574912 2024-03-25T00:34:11.866 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 55834574912 2024-03-25T00:34:11.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574912' 2024-03-25T00:34:11.867 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:34:12.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574912 -lt 55834574912 2024-03-25T00:34:12.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:12.275 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378679 2024-03-25T00:34:12.275 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:12.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:34:12.276 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378679 2024-03-25T00:34:12.277 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:12.278 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378679 2024-03-25T00:34:12.278 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 81604378679 2024-03-25T00:34:12.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378679' 2024-03-25T00:34:12.279 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:34:12.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378679 -lt 81604378679 2024-03-25T00:34:12.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:12.726 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117041 2024-03-25T00:34:12.726 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:12.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:34:12.728 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117041 2024-03-25T00:34:12.728 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:12.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117041 2024-03-25T00:34:12.730 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 115964117041 2024-03-25T00:34:12.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117041' 2024-03-25T00:34:12.730 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:34:13.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117041 -lt 115964117041 2024-03-25T00:34:13.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:13.196 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757288 2024-03-25T00:34:13.196 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:13.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:34:13.198 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757288 2024-03-25T00:34:13.198 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:13.200 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757288 2024-03-25T00:34:13.200 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 163208757288 2024-03-25T00:34:13.200 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757288' 2024-03-25T00:34:13.200 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:34:13.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757289 -lt 163208757288 2024-03-25T00:34:13.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:13.612 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-210453397539 2024-03-25T00:34:13.612 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:13.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:34:13.614 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-210453397539 2024-03-25T00:34:13.614 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:13.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397539 2024-03-25T00:34:13.616 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 210453397539 2024-03-25T00:34:13.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 210453397539' 2024-03-25T00:34:13.616 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:34:14.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397539 -lt 210453397539 2024-03-25T00:34:14.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:14.018 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-257698037790 2024-03-25T00:34:14.018 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:14.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:34:14.020 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-257698037790 2024-03-25T00:34:14.020 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:14.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037790 2024-03-25T00:34:14.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 257698037790' 2024-03-25T00:34:14.022 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 257698037790 2024-03-25T00:34:14.022 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:34:14.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037790 -lt 257698037790 2024-03-25T00:34:14.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:34:14.436 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-343597383691 2024-03-25T00:34:14.436 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:34:14.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-25T00:34:14.439 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-343597383691 2024-03-25T00:34:14.439 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:34:14.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383691 2024-03-25T00:34:14.441 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.8 seq 343597383691 2024-03-25T00:34:14.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 343597383691' 2024-03-25T00:34:14.441 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-25T00:34:14.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383691 -lt 343597383691 2024-03-25T00:34:14.852 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:34:14.853 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:34:14.853 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:34:15.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:34:15.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:34:15.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:34:15.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:34:15.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:34:15.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:34:15.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:34:15.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:34:15.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:34:15.793 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:34:15.793 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:34:15.793 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:34:16.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:34:16.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:34:16.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:34:16.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2024-03-25T00:34:16.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:34:16.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:34:16.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-03-25T00:34:16.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:34:16.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:34:16.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-03-25T00:34:16.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:34:16.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:34:16.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:445: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2024-03-25T00:34:16.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:34:16.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-03-25T00:34:16.926 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' does not exist 2024-03-25T00:34:16.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-03-25T00:34:17.548 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile myprofile does not exist 2024-03-25T00:34:17.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-03-25T00:34:17.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:34:17.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:34:17.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:34:17.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:34:17.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:34:17.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:34:17.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:34:17.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:34:17.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:34:17.881 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:34:17.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:34:17.883 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:34:17.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:34:17.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:34:17.886 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:34:17.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:34:17.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:34:17.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:34:17.888 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:34:17.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:34:17.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:34:17.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:34:17.940 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:34:17.940 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:17.940 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:17.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:34:17.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:34:17.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:34:17.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-03-25T00:34:17.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-03-25T00:34:17.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-03-25T00:34:17.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-03-25T00:34:17.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:34:17.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:34:17.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:34:17.943 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:34:17.943 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:34:17.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:34:17.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:34:17.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:34:17.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:34:17.945 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:34:17.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:34:17.946 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:34:17.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:34:17.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:34:17.949 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:34:17.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:34:17.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:34:17.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:34:17.951 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:34:17.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:34:17.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:34:17.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:34:17.954 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:34:17.954 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:17.954 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:17.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:34:17.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:34:17.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:34:17.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-03-25T00:34:17.956 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:34:17.957 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:17.957 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:17.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81310 2024-03-25T00:34:17.959 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:34:17.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:34:17.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:34:17.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-03-25T00:34:17.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-03-25T00:34:17.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-03-25T00:34:17.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:34:17.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:34:17.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:34:17.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-03-25T00:34:17.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-03-25T00:34:18.450 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:34:18.450 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:34:18.451 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:34:18.451 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:34:18.451 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:18.451 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:18.451 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:34:18.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T00:34:18.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:34:18.671 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:34:18.671 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:34:18.671 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:34:18.671 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:34:18.673 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:34:18.673 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:34:18.673 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:34:18.673 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:34:18.673 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:34:18.673 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:18.673 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:18.673 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:34:18.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:34:18.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get fsid 2024-03-25T00:34:18.791 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:34:18.791 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:34:18.791 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:34:18.791 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:34:18.791 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:34:18.791 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:34:18.791 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:34:18.792 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:34:18.792 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:34:18.792 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:18.792 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:18.792 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:34:18.792 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:34:18.792 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get mon_host 2024-03-25T00:34:18.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-03-25T00:34:18.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-03-25T00:34:18.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:34:18.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:34:18.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:34:18.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-03-25T00:34:18.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:34:19.797 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:34:19.797 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:34:19.797 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:34:19.798 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:34:19.798 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:19.798 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:19.798 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:34:19.799 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:34:19.800 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:34:19.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-03-25T00:34:19.833 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T00:34:20.124 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' already exists 2024-03-25T00:34:20.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:34:21.147 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-03-25T00:34:21.148 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:34:21.148 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:34:21.148 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:34:21.148 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:21.149 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:21.149 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:34:21.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-03-25T00:34:21.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-mon.a.asok log flush 2024-03-25T00:34:21.249 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:34:21.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-03-25T00:34:21.263 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:34:18.493+0000 7f99e5691c80 0 load: jerasure load: lrc 2024-03-25T00:34:21.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2024-03-25T00:34:21.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2024-03-25T00:34:21.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:452: TEST_ec_recovery_multiple_objects: local objname=myobject 2024-03-25T00:34:21.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:34:21.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:455: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:34:21.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_objects: setup_osds 7 2024-03-25T00:34:21.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-03-25T00:34:21.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-03-25T00:34:21.265 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-03-25T00:34:21.266 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-03-25T00:34:21.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:34:21.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-03-25T00:34:21.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:34:21.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:34:21.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:34:21.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:34:21.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-03-25T00:34:21.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:34:21.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:34:21.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:34:21.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:34:21.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-03-25T00:34:21.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-03-25T00:34:21.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:34:21.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:34:21.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:34:21.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:34:21.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:34:21.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:34:21.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:34:21.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:34:21.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-03-25T00:34:21.274 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:34:21.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4101836f-7479-4af2-81a4-05e8dd8a1ef7 2024-03-25T00:34:21.274 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 4101836f-7479-4af2-81a4-05e8dd8a1ef7 2024-03-25T00:34:21.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4101836f-7479-4af2-81a4-05e8dd8a1ef7' 2024-03-25T00:34:21.274 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:34:21.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNxgBmMN+NERAAKml6tvGwNgRsGb5pPVW99A== 2024-03-25T00:34:21.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNxgBmMN+NERAAKml6tvGwNgRsGb5pPVW99A=="}' 2024-03-25T00:34:21.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4101836f-7479-4af2-81a4-05e8dd8a1ef7 -i td/test-erasure-eio/0/new.json 2024-03-25T00:34:21.584 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:34:21.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-03-25T00:34:21.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNxgBmMN+NERAAKml6tvGwNgRsGb5pPVW99A== --osd-uuid 4101836f-7479-4af2-81a4-05e8dd8a1ef7 2024-03-25T00:34:21.644 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:21.641+0000 7ff62199e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:34:21.652 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:21.649+0000 7ff62199e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:34:21.653 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:21.649+0000 7ff62199e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:34:21.654 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:21.653+0000 7ff62199e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:34:21.654 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:21.653+0000 7ff62199e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:34:21.654 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:21.653+0000 7ff62199e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:34:21.692 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:21.689+0000 7ff62199e7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-03-25T00:34:27.628 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-03-25T00:34:27.628 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:34:27.630 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:34:27.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:34:27.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:34:28.186 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:34:28.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:34:28.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:34:28.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:34:28.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:34:28.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:34:28.224 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:28.221+0000 7f510abc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:34:28.267 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:28.265+0000 7f510abc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:34:28.271 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:28.265+0000 7f510abc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:34:28.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:34:28.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:34:28.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:34:28.577 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:34:28.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:34:28.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:34:28.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:28.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:34:28.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:28.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:28.816 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:28.813+0000 7f510abc77c0 -1 Falling back to public interface 2024-03-25T00:34:29.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:30.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:30.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:30.003 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:34:30.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:34:30.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:30.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:30.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:31.316 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:31.313+0000 7f510abc77c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:34:31.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:31.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:31.431 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:34:31.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:34:31.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:31.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:31.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:32.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:32.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:32.912 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:34:32.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:34:32.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:32.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:33.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:34.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:34.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:34.341 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:34:34.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:34:34.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:34.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:34.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:35.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:35.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:35.785 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:34:35.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:34:35.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:35.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:36.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:37.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:37.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:37.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:34:37.328 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:34:37.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:37.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:37.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:38.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:38.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:38.776 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:34:38.776 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:34:38.776 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:38.776 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:39.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:40.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:40.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:40.235 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:34:40.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:34:40.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:40.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:40.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:41.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:41.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:41.701 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:34:41.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:34:41.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:41.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:42.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:43.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:43.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:43.157 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:34:43.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:34:43.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:43.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:44.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:45.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:45.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:45.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:34:45.055 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:34:45.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:45.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:45.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:46.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:46.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:46.950 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:34:46.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:34:46.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:46.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:47.848 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:48.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:48.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:48.850 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:34:48.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:34:48.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:48.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:49.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:51.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:51.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:51.002 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:34:51.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:34:51.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:51.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:51.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:52.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:52.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:52.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:34:52.755 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:34:52.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:52.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:53.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:34:54.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:34:54.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:34:54.757 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:34:54.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:34:54.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:34:54.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:34:55.216 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3577990644,v1:127.0.0.1:6803/3577990644] [v2:127.0.0.1:6804/3577990644,v1:127.0.0.1:6805/3577990644] exists,up 4101836f-7479-4af2-81a4-05e8dd8a1ef7 2024-03-25T00:34:55.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:34:55.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:34:55.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:34:55.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:34:55.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-03-25T00:34:55.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:34:55.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:34:55.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T00:34:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:34:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:34:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:34:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:34:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:34:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:34:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:34:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:34:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:34:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:34:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:34:55.223 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:34:55.223 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:34:55.223 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:34:55.223 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:34:55.223 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:34:55.223 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:34:55.223 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:34:55.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:34:55.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:34:55.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:34:55.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:34:55.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:34:55.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:34:55.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:34:55.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:34:55.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:34:55.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:34:55.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:34:55.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:34:55.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:34:55.224 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:34:55.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=766c491a-ca70-4386-aa00-b698c5aa107c 2024-03-25T00:34:55.225 INFO:tasks.workunit.client.0.smithi080.stdout:add osd1 766c491a-ca70-4386-aa00-b698c5aa107c 2024-03-25T00:34:55.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 766c491a-ca70-4386-aa00-b698c5aa107c' 2024-03-25T00:34:55.226 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:34:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvxgBmUGfKDhAA5W8NJXqay4vjtx6Bw7tlfQ== 2024-03-25T00:34:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvxgBmUGfKDhAA5W8NJXqay4vjtx6Bw7tlfQ=="}' 2024-03-25T00:34:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 766c491a-ca70-4386-aa00-b698c5aa107c -i td/test-erasure-eio/1/new.json 2024-03-25T00:34:55.808 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:34:55.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-03-25T00:34:55.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvxgBmUGfKDhAA5W8NJXqay4vjtx6Bw7tlfQ== --osd-uuid 766c491a-ca70-4386-aa00-b698c5aa107c 2024-03-25T00:34:55.866 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:55.868+0000 7f8d7da6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:34:55.887 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:55.888+0000 7f8d7da6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:34:55.888 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:55.888+0000 7f8d7da6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:34:55.889 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:55.888+0000 7f8d7da6b7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:34:55.889 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:55.888+0000 7f8d7da6b7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:34:55.889 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:55.888+0000 7f8d7da6b7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:34:55.932 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:34:55.932+0000 7f8d7da6b7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-03-25T00:35:02.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-03-25T00:35:02.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:35:02.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T00:35:02.060 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd1 key to auth repository 2024-03-25T00:35:02.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:35:02.646 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.1 2024-03-25T00:35:02.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T00:35:02.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:35:02.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:35:02.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:35:02.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:35:02.683 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:02.684+0000 7f97425417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:35:02.701 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:02.700+0000 7f97425417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:35:02.702 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:02.704+0000 7f97425417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:35:03.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T00:35:03.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:35:03.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:35:03.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:35:03.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:35:03.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:03.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:35:03.110 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:35:03.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:03.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:03.264 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:03.264+0000 7f97425417c0 -1 Falling back to public interface 2024-03-25T00:35:03.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:04.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:04.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:04.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:35:04.519 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:35:04.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:04.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:04.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:05.781 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:05.780+0000 7f97425417c0 -1 osd.1 0 log_to_monitors true 2024-03-25T00:35:05.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:05.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:05.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:35:05.927 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:35:05.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:05.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:06.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:07.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:07.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:07.478 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:35:07.478 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:35:07.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:07.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:07.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:08.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:08.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:08.936 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:35:08.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:35:08.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:08.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:09.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:10.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:10.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:10.585 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:35:10.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:35:10.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:10.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:10.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:11.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:11.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:11.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:35:11.999 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:35:12.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:12.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:12.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:13.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:13.654 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:13.655 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:35:13.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:35:13.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:13.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:14.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:15.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:15.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:15.085 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:35:15.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:35:15.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:15.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:15.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:16.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:16.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:16.565 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:35:16.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:35:16.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:16.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:17.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:18.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:18.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:18.020 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:35:18.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:35:18.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:18.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:18.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:19.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:19.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:19.588 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:35:19.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:35:19.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:19.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:20.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:21.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:21.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:21.301 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:35:21.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:35:21.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:21.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:22.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:23.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:23.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:23.041 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:35:23.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:35:23.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:23.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:24.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:25.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:25.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:25.150 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:35:25.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:35:25.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:25.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:26.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:27.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:27.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:27.225 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:35:27.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:35:27.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:27.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:28.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:29.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:29.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:29.363 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:35:29.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:35:29.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:29.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:35:29.857 INFO:tasks.workunit.client.0.smithi080.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2827612633,v1:127.0.0.1:6811/2827612633] [v2:127.0.0.1:6812/2827612633,v1:127.0.0.1:6813/2827612633] exists,up 766c491a-ca70-4386-aa00-b698c5aa107c 2024-03-25T00:35:29.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:35:29.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:35:29.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:35:29.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:35:29.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-03-25T00:35:29.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:35:29.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:35:29.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:35:29.861 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:35:29.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:35:29.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:35:29.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:35:29.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:35:29.863 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:35:29.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=945bdfa2-0544-4b86-882e-89442d1991ea 2024-03-25T00:35:29.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 945bdfa2-0544-4b86-882e-89442d1991ea' 2024-03-25T00:35:29.864 INFO:tasks.workunit.client.0.smithi080.stdout:add osd2 945bdfa2-0544-4b86-882e-89442d1991ea 2024-03-25T00:35:29.864 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:35:29.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRxgBm7Xv9NBAA1XAmMK/XPmvJkS/+VLPofA== 2024-03-25T00:35:29.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRxgBm7Xv9NBAA1XAmMK/XPmvJkS/+VLPofA=="}' 2024-03-25T00:35:29.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 945bdfa2-0544-4b86-882e-89442d1991ea -i td/test-erasure-eio/2/new.json 2024-03-25T00:35:30.379 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:35:30.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-03-25T00:35:30.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRxgBm7Xv9NBAA1XAmMK/XPmvJkS/+VLPofA== --osd-uuid 945bdfa2-0544-4b86-882e-89442d1991ea 2024-03-25T00:35:30.440 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:30.440+0000 7ff0f34e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:35:30.600 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:30.600+0000 7ff0f34e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:35:30.602 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:30.600+0000 7ff0f34e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:35:30.602 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:30.604+0000 7ff0f34e37c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:35:30.603 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:30.604+0000 7ff0f34e37c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:35:30.603 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:30.604+0000 7ff0f34e37c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:35:30.715 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:30.716+0000 7ff0f34e37c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-03-25T00:35:36.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-03-25T00:35:36.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:35:36.521 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd2 key to auth repository 2024-03-25T00:35:36.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T00:35:36.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:35:37.114 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:35:37.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T00:35:37.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:35:37.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:35:37.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:35:37.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:35:37.144 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:37.143+0000 7fd3311f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:35:37.158 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:37.159+0000 7fd3311f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:35:37.159 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:37.159+0000 7fd3311f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:35:37.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T00:35:37.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:35:37.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:35:37.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:35:37.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:35:37.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:37.523 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:35:37.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:35:37.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:37.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:37.747 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:37.747+0000 7fd3311f97c0 -1 Falling back to public interface 2024-03-25T00:35:37.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:38.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:38.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:38.941 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:35:38.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:35:38.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:38.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:39.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:40.231 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:35:40.231+0000 7fd3311f97c0 -1 osd.2 0 log_to_monitors true 2024-03-25T00:35:40.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:40.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:40.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:35:40.367 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:35:40.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:40.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:40.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:41.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:41.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:41.954 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:35:41.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:35:41.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:41.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:42.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:43.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:43.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:43.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:35:43.391 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:35:43.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:43.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:43.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:44.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:44.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:44.817 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:35:44.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:35:44.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:44.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:45.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:46.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:46.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:46.500 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:35:46.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:35:46.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:46.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:47.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:48.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:48.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:48.045 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:35:48.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:35:48.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:48.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:48.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:49.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:49.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:49.491 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:35:49.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:35:49.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:49.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:50.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:51.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:51.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:51.028 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:35:51.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:35:51.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:51.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:51.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:52.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:52.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:52.490 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:35:52.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:35:52.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:52.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:53.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:54.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:54.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:54.075 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:35:54.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:35:54.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:54.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:54.662 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:55.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:55.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:55.664 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:35:55.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:35:55.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:55.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:56.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:57.412 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:57.412 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:57.412 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:35:57.413 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:35:57.413 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:57.413 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:58.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:35:59.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:35:59.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:35:59.165 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:35:59.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:35:59.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:35:59.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:35:59.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:00.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:00.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:00.855 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:36:00.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:36:00.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:00.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:36:01.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:02.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:02.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:02.363 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:36:02.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:36:02.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:02.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:36:03.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:04.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:04.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:04.214 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:36:04.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:36:04.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:04.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:36:04.848 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1998152581,v1:127.0.0.1:6819/1998152581] [v2:127.0.0.1:6820/1998152581,v1:127.0.0.1:6821/1998152581] exists,up 945bdfa2-0544-4b86-882e-89442d1991ea 2024-03-25T00:36:04.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:36:04.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:36:04.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:36:04.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:36:04.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-03-25T00:36:04.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:36:04.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:36:04.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T00:36:04.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:36:04.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:36:04.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:36:04.851 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:36:04.852 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:36:04.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:36:04.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:36:04.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:36:04.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:36:04.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:36:04.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:36:04.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:36:04.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:36:04.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:36:04.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:36:04.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:36:04.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:36:04.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:36:04.854 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:36:04.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dc9f2eee-4e72-406c-bb06-830720774e90 2024-03-25T00:36:04.855 INFO:tasks.workunit.client.0.smithi080.stdout:add osd3 dc9f2eee-4e72-406c-bb06-830720774e90 2024-03-25T00:36:04.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 dc9f2eee-4e72-406c-bb06-830720774e90' 2024-03-25T00:36:04.856 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:36:04.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0xgBmoHKoNBAAJQw2ud464JMxTiw0xKWUvw== 2024-03-25T00:36:04.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0xgBmoHKoNBAAJQw2ud464JMxTiw0xKWUvw=="}' 2024-03-25T00:36:04.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dc9f2eee-4e72-406c-bb06-830720774e90 -i td/test-erasure-eio/3/new.json 2024-03-25T00:36:05.457 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:36:05.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-03-25T00:36:05.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0xgBmoHKoNBAAJQw2ud464JMxTiw0xKWUvw== --osd-uuid dc9f2eee-4e72-406c-bb06-830720774e90 2024-03-25T00:36:05.519 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:05.519+0000 7facc15b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:05.566 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:05.563+0000 7facc15b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:05.567 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:05.567+0000 7facc15b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:05.568 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:05.567+0000 7facc15b87c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:36:05.568 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:05.567+0000 7facc15b87c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:36:05.568 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:05.567+0000 7facc15b87c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:36:05.674 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:05.675+0000 7facc15b87c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-03-25T00:36:11.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-03-25T00:36:11.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:36:11.890 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd3 key to auth repository 2024-03-25T00:36:11.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T00:36:11.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:36:12.447 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.3 2024-03-25T00:36:12.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T00:36:12.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:36:12.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:36:12.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:36:12.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:36:12.483 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:12.483+0000 7ff4de54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:12.518 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:12.515+0000 7ff4de54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:12.520 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:12.519+0000 7ff4de54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:12.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T00:36:12.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:36:12.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:36:12.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:36:12.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:36:12.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:12.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:36:12.898 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:36:12.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:12.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:13.076 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:13.075+0000 7ff4de54f7c0 -1 Falling back to public interface 2024-03-25T00:36:13.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:14.318 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:14.318 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:14.318 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:36:14.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:36:14.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:14.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:14.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:15.632 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:15.631+0000 7ff4de54f7c0 -1 osd.3 0 log_to_monitors true 2024-03-25T00:36:15.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:15.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:15.723 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:36:15.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:36:15.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:15.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:16.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:17.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:17.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:17.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:36:17.363 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:36:17.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:17.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:18.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:19.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:19.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:19.068 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:36:19.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:36:19.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:19.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:19.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:20.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:20.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:20.659 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:36:20.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:36:20.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:20.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:21.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:22.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:22.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:22.178 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:36:22.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:36:22.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:22.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:22.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:23.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:23.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:23.730 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:36:23.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:36:23.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:23.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:24.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:25.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:25.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:25.342 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:36:25.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:36:25.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:25.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:25.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:26.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:26.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:26.907 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:36:26.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:36:26.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:26.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:27.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:28.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:28.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:28.501 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:36:28.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:36:28.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:28.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:28.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:29.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:29.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:29.993 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:36:29.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:36:29.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:29.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:30.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:31.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:31.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:31.819 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:36:31.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:36:31.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:31.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:32.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:33.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:33.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:33.684 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:36:33.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:36:33.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:33.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:34.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:35.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:35.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:35.458 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:36:35.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:36:35.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:35.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:36.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:37.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:37.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:37.230 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:36:37.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:36:37.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:37.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:37.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:38.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:38.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:38.939 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:36:38.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:36:38.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:38.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:39.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:40.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:40.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:40.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:36:40.472 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:36:40.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:40.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:36:40.980 INFO:tasks.workunit.client.0.smithi080.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1544202883,v1:127.0.0.1:6827/1544202883] [v2:127.0.0.1:6828/1544202883,v1:127.0.0.1:6829/1544202883] exists,up dc9f2eee-4e72-406c-bb06-830720774e90 2024-03-25T00:36:40.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:36:40.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:36:40.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:36:40.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:36:40.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-03-25T00:36:40.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:36:40.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:36:40.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-25T00:36:40.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:36:40.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-03-25T00:36:40.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:36:40.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:36:40.986 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:36:40.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-03-25T00:36:40.988 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:36:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8b9c34b-6a1f-4115-b4ab-8b5132eaedbc 2024-03-25T00:36:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 c8b9c34b-6a1f-4115-b4ab-8b5132eaedbc' 2024-03-25T00:36:40.988 INFO:tasks.workunit.client.0.smithi080.stdout:add osd4 c8b9c34b-6a1f-4115-b4ab-8b5132eaedbc 2024-03-25T00:36:40.989 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:36:41.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZxwBmm/qOABAAache2RQ/JJy4/4FWagEUCA== 2024-03-25T00:36:41.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZxwBmm/qOABAAache2RQ/JJy4/4FWagEUCA=="}' 2024-03-25T00:36:41.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8b9c34b-6a1f-4115-b4ab-8b5132eaedbc -i td/test-erasure-eio/4/new.json 2024-03-25T00:36:41.624 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:36:41.641 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-03-25T00:36:41.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZxwBmm/qOABAAache2RQ/JJy4/4FWagEUCA== --osd-uuid c8b9c34b-6a1f-4115-b4ab-8b5132eaedbc 2024-03-25T00:36:41.685 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:41.682+0000 7f92c19417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:41.773 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:41.770+0000 7f92c19417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:41.775 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:41.774+0000 7f92c19417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:41.775 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:41.774+0000 7f92c19417c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:36:41.775 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:41.774+0000 7f92c19417c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:36:41.775 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:41.774+0000 7f92c19417c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:36:41.846 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:41.842+0000 7f92c19417c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-03-25T00:36:48.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-03-25T00:36:48.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:36:48.741 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd4 key to auth repository 2024-03-25T00:36:48.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-25T00:36:48.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:36:49.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-25T00:36:49.308 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.4 2024-03-25T00:36:49.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:36:49.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:36:49.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:36:49.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:36:49.344 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:49.342+0000 7f651577f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:49.355 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:49.354+0000 7f651577f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:49.356 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:49.354+0000 7f651577f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:36:49.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-25T00:36:49.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:36:49.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-25T00:36:49.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:36:49.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:36:49.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:49.727 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:36:49.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:36:49.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:49.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:36:49.949 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:49.946+0000 7f651577f7c0 -1 Falling back to public interface 2024-03-25T00:36:50.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:51.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:51.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:51.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:36:51.142 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:36:51.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:51.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:36:51.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:52.455 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:36:52.454+0000 7f651577f7c0 -1 osd.4 0 log_to_monitors true 2024-03-25T00:36:52.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:52.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:52.561 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:36:52.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:36:52.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:52.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:36:53.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:54.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:54.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:54.276 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:36:54.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:36:54.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:54.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:36:54.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:55.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:55.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:55.844 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:36:55.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:36:55.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:55.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:36:56.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:57.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:57.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:57.396 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:36:57.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:36:57.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:57.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:36:57.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:36:58.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:36:58.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:36:58.912 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:36:58.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:36:58.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:36:58.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:36:59.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:00.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:00.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:00.354 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:37:00.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:37:00.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:00.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:00.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:01.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:01.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:01.885 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:37:01.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:37:01.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:01.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:03.374 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:03.374 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:03.375 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:37:03.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:37:03.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:03.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:03.896 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:04.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:04.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:04.899 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:37:04.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:37:04.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:04.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:05.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:06.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:06.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:06.329 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:37:06.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:37:06.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:06.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:06.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:07.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:07.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:07.787 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:37:07.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:37:07.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:07.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:08.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:09.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:09.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:09.588 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:37:09.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:37:09.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:09.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:10.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:11.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:11.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:11.502 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:37:11.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:37:11.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:11.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:12.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:13.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:13.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:13.456 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:37:13.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:37:13.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:13.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:14.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:15.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:15.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:15.204 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:37:15.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:37:15.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:15.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:16.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:17.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:17.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:17.573 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:37:17.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:37:17.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:17.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:18.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:19.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:19.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:19.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:37:19.005 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:37:19.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:19.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:37:19.426 INFO:tasks.workunit.client.0.smithi080.stdout:osd.4 up in weight 1 up_from 36 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2759365876,v1:127.0.0.1:6835/2759365876] [v2:127.0.0.1:6836/2759365876,v1:127.0.0.1:6837/2759365876] exists,up c8b9c34b-6a1f-4115-b4ab-8b5132eaedbc 2024-03-25T00:37:19.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:37:19.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:37:19.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:37:19.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:37:19.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-03-25T00:37:19.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:37:19.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:37:19.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-25T00:37:19.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:37:19.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-03-25T00:37:19.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:37:19.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:37:19.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:37:19.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:37:19.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-03-25T00:37:19.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-03-25T00:37:19.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:37:19.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:37:19.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:37:19.434 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:37:19.434 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:37:19.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:37:19.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:37:19.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:37:19.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:37:19.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:37:19.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-03-25T00:37:19.436 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:37:19.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3acb7926-11c5-488f-bab6-288a1c45ce05 2024-03-25T00:37:19.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 3acb7926-11c5-488f-bab6-288a1c45ce05' 2024-03-25T00:37:19.436 INFO:tasks.workunit.client.0.smithi080.stdout:add osd5 3acb7926-11c5-488f-bab6-288a1c45ce05 2024-03-25T00:37:19.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:37:19.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/xwBmm/U/GxAALVgU0ZH5ttDcvFsmAnMLqg== 2024-03-25T00:37:19.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/xwBmm/U/GxAALVgU0ZH5ttDcvFsmAnMLqg=="}' 2024-03-25T00:37:19.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3acb7926-11c5-488f-bab6-288a1c45ce05 -i td/test-erasure-eio/5/new.json 2024-03-25T00:37:20.397 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:37:20.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-03-25T00:37:20.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/xwBmm/U/GxAALVgU0ZH5ttDcvFsmAnMLqg== --osd-uuid 3acb7926-11c5-488f-bab6-288a1c45ce05 2024-03-25T00:37:20.462 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:20.457+0000 7f43c9b887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:37:20.542 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:20.541+0000 7f43c9b887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:37:20.544 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:20.541+0000 7f43c9b887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:37:20.544 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:20.541+0000 7f43c9b887c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:37:20.545 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:20.541+0000 7f43c9b887c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:37:20.545 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:20.541+0000 7f43c9b887c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:37:20.678 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:20.673+0000 7f43c9b887c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-03-25T00:37:27.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-03-25T00:37:27.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:37:27.950 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd5 key to auth repository 2024-03-25T00:37:27.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-25T00:37:27.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:37:28.501 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.5 2024-03-25T00:37:28.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-25T00:37:28.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:37:28.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:37:28.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:37:28.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:37:28.538 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:28.533+0000 7fd4d0c537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:37:28.542 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:28.537+0000 7fd4d0c537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:37:28.543 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:28.541+0000 7fd4d0c537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:37:28.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-25T00:37:28.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:37:28.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T00:37:28.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:37:28.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:37:28.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:28.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:37:28.976 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:37:28.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:28.977 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:29.120 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:29.117+0000 7fd4d0c537c0 -1 Falling back to public interface 2024-03-25T00:37:29.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:30.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:30.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:30.386 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:37:30.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:37:30.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:30.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:30.801 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:31.685 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:31.681+0000 7fd4d0c537c0 -1 osd.5 0 log_to_monitors true 2024-03-25T00:37:31.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:31.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:31.803 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:37:31.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:37:31.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:31.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:32.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:33.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:33.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:33.383 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:37:33.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:37:33.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:33.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:34.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:35.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:35.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:35.160 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:37:35.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:37:35.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:35.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:35.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:36.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:36.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:36.742 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:37:36.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:37:36.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:36.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:37.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:38.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:38.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:38.213 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:37:38.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:37:38.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:38.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:38.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:39.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:39.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:39.735 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:37:39.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:37:39.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:39.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:40.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:41.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:41.182 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:37:41.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:41.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:37:41.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:41.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:41.703 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:42.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:42.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:42.705 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:37:42.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:37:42.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:42.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:43.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:44.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:44.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:44.254 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:37:44.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:37:44.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:44.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:44.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:45.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:45.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:45.881 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:37:45.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:37:45.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:45.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:46.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:47.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:47.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:47.734 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:37:47.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:37:47.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:47.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:48.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:49.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:49.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:49.469 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:37:49.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:37:49.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:49.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:50.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:51.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:51.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:51.286 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:37:51.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:37:51.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:51.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:51.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:52.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:52.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:52.821 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:37:52.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:37:52.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:52.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:53.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:54.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:54.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:54.983 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:37:54.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:37:54.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:54.984 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:55.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:56.504 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:56.504 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:56.504 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:37:56.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:37:56.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:56.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:56.978 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:37:57.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:37:57.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:37:57.980 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:37:57.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:37:57.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:37:57.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:37:58.756 INFO:tasks.workunit.client.0.smithi080.stdout:osd.5 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3806334532,v1:127.0.0.1:6843/3806334532] [v2:127.0.0.1:6844/3806334532,v1:127.0.0.1:6845/3806334532] exists,up 3acb7926-11c5-488f-bab6-288a1c45ce05 2024-03-25T00:37:58.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:37:58.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:37:58.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:37:58.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-03-25T00:37:58.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:37:58.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:37:58.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:37:58.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:37:58.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:37:58.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:37:58.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:37:58.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:37:58.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:37:58.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-03-25T00:37:58.764 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:37:58.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cba7d6f2-9741-4664-a226-02fed65ef623 2024-03-25T00:37:58.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 cba7d6f2-9741-4664-a226-02fed65ef623' 2024-03-25T00:37:58.764 INFO:tasks.workunit.client.0.smithi080.stdout:add osd6 cba7d6f2-9741-4664-a226-02fed65ef623 2024-03-25T00:37:58.765 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:37:58.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBmxwBmoNcgLxAASFF6CzVwsVvLA94/F8ZhQw== 2024-03-25T00:37:58.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBmxwBmoNcgLxAASFF6CzVwsVvLA94/F8ZhQw=="}' 2024-03-25T00:37:58.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cba7d6f2-9741-4664-a226-02fed65ef623 -i td/test-erasure-eio/6/new.json 2024-03-25T00:37:59.425 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:37:59.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-03-25T00:37:59.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBmxwBmoNcgLxAASFF6CzVwsVvLA94/F8ZhQw== --osd-uuid cba7d6f2-9741-4664-a226-02fed65ef623 2024-03-25T00:37:59.489 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:59.476+0000 7fd3b795c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:37:59.529 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:59.528+0000 7fd3b795c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:37:59.531 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:59.532+0000 7fd3b795c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:37:59.531 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:59.532+0000 7fd3b795c7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:37:59.531 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:59.532+0000 7fd3b795c7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:37:59.532 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:59.532+0000 7fd3b795c7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:37:59.583 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:37:59.584+0000 7fd3b795c7c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-03-25T00:38:05.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-03-25T00:38:05.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:38:05.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-25T00:38:05.996 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd6 key to auth repository 2024-03-25T00:38:05.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:38:06.593 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.6 2024-03-25T00:38:06.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-25T00:38:06.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:38:06.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:38:06.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:38:06.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:38:06.629 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:06.628+0000 7f8e95cf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:38:06.634 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:06.632+0000 7f8e95cf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:38:06.635 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:06.636+0000 7f8e95cf97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:38:07.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-25T00:38:07.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:38:07.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-25T00:38:07.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:38:07.040 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:38:07.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:38:07.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:07.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:38:07.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:07.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:07.204 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:07.204+0000 7f8e95cf97c0 -1 Falling back to public interface 2024-03-25T00:38:07.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:08.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:08.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:08.475 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:38:08.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:38:08.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:08.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:08.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:09.722 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:09.720+0000 7f8e95cf97c0 -1 osd.6 0 log_to_monitors true 2024-03-25T00:38:09.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:09.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:09.883 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:38:09.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:38:09.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:09.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:10.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:11.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:11.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:11.440 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:38:11.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:38:11.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:11.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:12.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:13.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:13.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:13.203 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:38:13.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:38:13.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:13.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:13.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:14.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:14.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:14.721 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:38:14.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:38:14.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:14.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:15.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:16.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:16.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:16.230 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:38:16.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:38:16.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:16.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:16.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:17.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:17.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:17.665 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:38:17.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:38:17.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:17.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:18.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:19.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:19.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:19.092 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:38:19.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:38:19.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:19.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:19.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:20.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:20.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:20.569 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:38:20.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:38:20.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:20.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:21.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:22.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:22.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:22.016 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:38:22.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:38:22.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:22.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:22.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:23.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:23.650 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:23.651 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:38:23.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:38:23.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:23.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:24.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:25.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:25.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:25.145 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:38:25.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:38:25.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:25.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:26.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:27.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:27.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:27.337 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:38:27.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:38:27.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:27.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:28.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:29.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:29.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:29.047 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:38:29.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:38:29.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:29.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:29.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:30.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:30.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:30.756 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:38:30.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:38:30.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:30.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:31.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:32.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:32.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:32.819 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:38:32.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:38:32.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:32.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:33.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:38:34.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:38:34.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:38:34.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:38:34.680 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:38:34.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:38:34.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:38:35.301 INFO:tasks.workunit.client.0.smithi080.stdout:osd.6 up in weight 1 up_from 54 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/329048112,v1:127.0.0.1:6851/329048112] [v2:127.0.0.1:6852/329048112,v1:127.0.0.1:6853/329048112] exists,up cba7d6f2-9741-4664-a226-02fed65ef623 2024-03-25T00:38:35.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:38:35.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:38:35.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:38:35.302 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-03-25T00:38:35.302 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:38:35.302 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:38:35.303 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:38:35.303 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:38:35.303 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:38:35.303 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.0.asok 2024-03-25T00:38:35.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-03-25T00:38:35.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.0.asok log flush 2024-03-25T00:38:35.410 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:38:35.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-03-25T00:38:35.427 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:34:28.821+0000 7f510abc77c0 0 load: jerasure load: lrc 2024-03-25T00:38:35.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:38:35.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:459: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2024-03-25T00:38:35.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:460: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2024-03-25T00:38:35.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:38:35.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-03-25T00:38:35.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-03-25T00:38:35.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-03-25T00:38:35.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-03-25T00:38:35.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-03-25T00:38:35.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-03-25T00:38:36.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-03-25T00:38:36.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-03-25T00:38:37.602 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' already exists 2024-03-25T00:38:37.618 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:38:38.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-03-25T00:38:38.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:38:38.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:38:38.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:38:38.620 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:38:38.621 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:38:38.621 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:38:38.622 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:38:38.622 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:38:38.622 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:38:38.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:38:38.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:38:38.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:38:38.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:38:38.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:38:38.731 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:38:39.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:38:39.291 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:38:39.291 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:38:39.291 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:38:39.291 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:38:39.291 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:38:39.292 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:38:39.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:38:39.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:39.292 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:38:39.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803826 2024-03-25T00:38:39.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803826 2024-03-25T00:38:39.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826' 2024-03-25T00:38:39.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:39.529 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:38:39.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574892 2024-03-25T00:38:39.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574892 2024-03-25T00:38:39.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-55834574892' 2024-03-25T00:38:39.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:39.752 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:38:39.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378659 2024-03-25T00:38:39.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378659 2024-03-25T00:38:39.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-55834574892 2-81604378659' 2024-03-25T00:38:39.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:39.917 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:38:40.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117025 2024-03-25T00:38:40.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117025 2024-03-25T00:38:40.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-55834574892 2-81604378659 3-115964117025' 2024-03-25T00:38:40.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:40.075 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:38:40.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822678 2024-03-25T00:38:40.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822678 2024-03-25T00:38:40.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-55834574892 2-81604378659 3-115964117025 4-154618822678' 2024-03-25T00:38:40.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:40.234 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:38:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528336 2024-03-25T00:38:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528336 2024-03-25T00:38:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-55834574892 2-81604378659 3-115964117025 4-154618822678 5-193273528336' 2024-03-25T00:38:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:40.387 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:38:40.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233995 2024-03-25T00:38:40.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233995 2024-03-25T00:38:40.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-55834574892 2-81604378659 3-115964117025 4-154618822678 5-193273528336 6-231928233995' 2024-03-25T00:38:40.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:40.563 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803826 2024-03-25T00:38:40.563 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:40.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:38:40.564 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803826 2024-03-25T00:38:40.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:40.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803826 2024-03-25T00:38:40.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-03-25T00:38:40.566 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803826 2024-03-25T00:38:40.566 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:38:40.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803827 -lt 25769803826 2024-03-25T00:38:40.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:40.965 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574892 2024-03-25T00:38:40.965 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:40.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:38:40.970 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574892 2024-03-25T00:38:40.970 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:40.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574892 2024-03-25T00:38:40.970 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 55834574892 2024-03-25T00:38:40.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574892' 2024-03-25T00:38:40.971 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:38:41.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574892 -lt 55834574892 2024-03-25T00:38:41.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:41.390 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378659 2024-03-25T00:38:41.390 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:41.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:38:41.392 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378659 2024-03-25T00:38:41.392 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:41.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378659 2024-03-25T00:38:41.394 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 81604378659 2024-03-25T00:38:41.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378659' 2024-03-25T00:38:41.394 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:38:41.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378659 -lt 81604378659 2024-03-25T00:38:41.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:41.789 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:41.790 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117025 2024-03-25T00:38:41.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:38:41.791 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117025 2024-03-25T00:38:41.791 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:41.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117025 2024-03-25T00:38:41.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117025' 2024-03-25T00:38:41.792 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 115964117025 2024-03-25T00:38:41.793 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:38:42.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117025 -lt 115964117025 2024-03-25T00:38:42.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:42.185 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-154618822678 2024-03-25T00:38:42.185 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:42.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:38:42.187 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-154618822678 2024-03-25T00:38:42.187 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:42.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822678 2024-03-25T00:38:42.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 154618822678' 2024-03-25T00:38:42.189 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 154618822678 2024-03-25T00:38:42.190 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:38:42.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822678 -lt 154618822678 2024-03-25T00:38:42.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:42.616 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-193273528336 2024-03-25T00:38:42.617 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:42.618 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:38:42.619 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-193273528336 2024-03-25T00:38:42.619 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:42.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528336 2024-03-25T00:38:42.621 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 193273528336 2024-03-25T00:38:42.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 193273528336' 2024-03-25T00:38:42.621 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:38:43.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528336 -lt 193273528336 2024-03-25T00:38:43.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:43.079 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-231928233995 2024-03-25T00:38:43.079 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:43.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:38:43.081 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-231928233995 2024-03-25T00:38:43.081 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:43.082 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233995 2024-03-25T00:38:43.083 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 231928233995 2024-03-25T00:38:43.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 231928233995' 2024-03-25T00:38:43.083 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:38:43.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233995 -lt 231928233995 2024-03-25T00:38:43.484 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:38:43.484 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:38:43.485 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:38:44.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:38:44.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:38:44.033 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:38:44.033 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:38:44.033 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:38:44.033 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:38:44.033 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:38:44.034 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:38:44.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:38:44.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:38:44.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:38:44.437 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:38:44.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:38:44.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:38:44.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:38:44.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:462: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2024-03-25T00:38:44.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:38:44.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:38:44.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-03-25T00:38:44.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:44.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:38:44.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:44.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:38:44.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:44.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:38:44.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:44.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:38:44.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-03-25T00:38:45.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2024-03-25T00:38:45.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:38:45.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:38:45.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-03-25T00:38:45.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:45.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:38:45.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:45.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:38:45.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:45.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:38:45.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:45.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:38:45.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-03-25T00:38:45.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:464: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2024-03-25T00:38:45.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:38:45.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:38:45.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-03-25T00:38:45.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:45.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:38:45.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:45.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:38:45.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:45.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:38:45.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:38:45.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:38:45.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-03-25T00:38:45.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:466: TEST_ec_recovery_multiple_objects: ceph osd out 0 2024-03-25T00:38:46.320 INFO:tasks.workunit.client.0.smithi080.stderr:osd.0 is already out. 2024-03-25T00:38:46.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:469: TEST_ec_recovery_multiple_objects: wait_for_clean 2024-03-25T00:38:46.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:38:46.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:38:46.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:38:46.337 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:38:46.338 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:38:46.338 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:38:46.339 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:38:46.339 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:38:46.339 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:38:46.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:38:46.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:38:46.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:38:46.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:38:46.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:38:46.443 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:38:47.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:38:47.145 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:38:47.145 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:38:47.145 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:38:47.145 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:38:47.145 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:38:47.145 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:38:47.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:38:47.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:47.145 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:38:47.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803830 2024-03-25T00:38:47.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803830 2024-03-25T00:38:47.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830' 2024-03-25T00:38:47.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:47.427 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:38:47.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574895 2024-03-25T00:38:47.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574895 2024-03-25T00:38:47.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574895' 2024-03-25T00:38:47.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:47.578 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:38:47.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378662 2024-03-25T00:38:47.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378662 2024-03-25T00:38:47.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574895 2-81604378662' 2024-03-25T00:38:47.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:47.852 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:38:47.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117029 2024-03-25T00:38:47.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117029 2024-03-25T00:38:48.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574895 2-81604378662 3-115964117029' 2024-03-25T00:38:48.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:48.000 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:38:48.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822681 2024-03-25T00:38:48.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822681 2024-03-25T00:38:48.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574895 2-81604378662 3-115964117029 4-154618822681' 2024-03-25T00:38:48.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:48.284 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:38:48.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528340 2024-03-25T00:38:48.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528340 2024-03-25T00:38:48.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574895 2-81604378662 3-115964117029 4-154618822681 5-193273528340' 2024-03-25T00:38:48.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:38:48.438 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:38:48.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233998 2024-03-25T00:38:48.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233998 2024-03-25T00:38:48.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-55834574895 2-81604378662 3-115964117029 4-154618822681 5-193273528340 6-231928233998' 2024-03-25T00:38:48.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:48.591 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803830 2024-03-25T00:38:48.591 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:48.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:38:48.592 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803830 2024-03-25T00:38:48.592 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:48.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803830 2024-03-25T00:38:48.593 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803830 2024-03-25T00:38:48.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-03-25T00:38:48.594 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:38:49.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803830 -lt 25769803830 2024-03-25T00:38:49.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:49.096 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574895 2024-03-25T00:38:49.096 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:49.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:38:49.099 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574895 2024-03-25T00:38:49.099 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:49.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574895 2024-03-25T00:38:49.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574895' 2024-03-25T00:38:49.101 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 55834574895 2024-03-25T00:38:49.101 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:38:49.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574895 -lt 55834574895 2024-03-25T00:38:49.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:49.515 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378662 2024-03-25T00:38:49.515 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:49.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:38:49.517 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378662 2024-03-25T00:38:49.518 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:49.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378662 2024-03-25T00:38:49.519 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 81604378662 2024-03-25T00:38:49.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378662' 2024-03-25T00:38:49.520 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:38:49.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378662 -lt 81604378662 2024-03-25T00:38:49.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:49.946 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117029 2024-03-25T00:38:49.946 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:49.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:38:49.949 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117029 2024-03-25T00:38:49.949 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:49.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117029 2024-03-25T00:38:49.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117029' 2024-03-25T00:38:49.950 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 115964117029 2024-03-25T00:38:49.950 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:38:50.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117029 -lt 115964117029 2024-03-25T00:38:50.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:50.371 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-154618822681 2024-03-25T00:38:50.371 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:50.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:38:50.374 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-154618822681 2024-03-25T00:38:50.374 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:50.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822681 2024-03-25T00:38:50.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 154618822681' 2024-03-25T00:38:50.376 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 154618822681 2024-03-25T00:38:50.376 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:38:50.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822682 -lt 154618822681 2024-03-25T00:38:50.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:50.798 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-193273528340 2024-03-25T00:38:50.798 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:50.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:38:50.799 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-193273528340 2024-03-25T00:38:50.799 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:50.800 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528340 2024-03-25T00:38:50.800 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 193273528340 2024-03-25T00:38:50.801 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 193273528340' 2024-03-25T00:38:50.801 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:38:51.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528340 -lt 193273528340 2024-03-25T00:38:51.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:38:51.205 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-231928233998 2024-03-25T00:38:51.205 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:38:51.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:38:51.209 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-231928233998 2024-03-25T00:38:51.210 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:38:51.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233998 2024-03-25T00:38:51.210 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 231928233998 2024-03-25T00:38:51.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 231928233998' 2024-03-25T00:38:51.211 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:38:51.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233999 -lt 231928233998 2024-03-25T00:38:51.631 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:38:51.631 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:38:51.631 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:38:52.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:38:52.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:38:52.172 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:38:52.172 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:38:52.172 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:38:52.172 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:38:52.173 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:38:52.173 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:38:52.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:38:52.588 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:38:52.589 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:38:52.589 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:38:53.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:38:53.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:38:53.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:38:53.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2024-03-25T00:38:53.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:38:53.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:38:53.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-03-25T00:38:53.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:38:53.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:38:53.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-03-25T00:38:53.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:38:53.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:38:53.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2024-03-25T00:38:53.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:38:53.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:38:53.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-03-25T00:38:53.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:38:53.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:38:53.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-03-25T00:38:53.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:38:53.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:38:53.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:473: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2024-03-25T00:38:53.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:38:53.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:38:53.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-03-25T00:38:53.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:38:53.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:38:53.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-03-25T00:38:53.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:38:53.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:38:53.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2024-03-25T00:38:53.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:38:53.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-03-25T00:38:53.733 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' does not exist 2024-03-25T00:38:53.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-03-25T00:38:54.500 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile myprofile does not exist 2024-03-25T00:38:54.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-03-25T00:38:54.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:38:54.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:38:54.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:38:54.519 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:38:54.519 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:38:54.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:38:54.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:38:54.519 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:38:54.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:38:54.835 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:38:54.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:38:54.836 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:38:54.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:38:54.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:38:54.838 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:38:54.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:38:54.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:38:54.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:38:54.840 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:38:54.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:38:54.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:38:54.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:38:54.881 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:38:54.881 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:38:54.882 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:38:54.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:38:54.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:38:54.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:38:54.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:38:54.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:38:54.886 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:38:54.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:38:54.887 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:38:54.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:38:54.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:38:54.889 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:38:54.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:38:54.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:38:54.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:38:54.890 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:38:54.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:38:54.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:38:54.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:38:54.893 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:38:54.894 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:38:54.894 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:38:54.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:38:54.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:38:54.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:38:54.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-03-25T00:38:54.896 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:38:54.896 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:38:54.896 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:38:54.896 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81310 2024-03-25T00:38:54.897 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:38:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:38:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:38:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-03-25T00:38:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-03-25T00:38:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-03-25T00:38:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:38:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:38:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:38:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-03-25T00:38:54.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-03-25T00:38:55.361 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:38:55.362 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:38:55.362 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:38:55.362 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:38:55.362 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:38:55.362 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:38:55.362 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:38:55.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T00:38:55.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:38:55.557 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:38:55.557 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:38:55.557 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:38:55.557 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:38:55.558 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:38:55.558 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:38:55.558 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:38:55.558 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:38:55.558 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:38:55.559 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:38:55.559 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:38:55.560 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:38:55.560 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:38:55.560 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get fsid 2024-03-25T00:38:55.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:38:55.675 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:38:55.676 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:38:55.676 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:38:55.676 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:38:55.676 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:38:55.676 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:38:55.676 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:38:55.677 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:38:55.677 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:38:55.677 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:38:55.677 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:38:55.677 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:38:55.677 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get mon_host 2024-03-25T00:38:55.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-03-25T00:38:55.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-03-25T00:38:55.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:38:55.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:38:55.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:38:55.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-03-25T00:38:55.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:38:56.701 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:38:56.701 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:38:56.701 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:38:56.702 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:38:56.702 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:38:56.702 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:38:56.702 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:38:56.703 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:38:56.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:38:56.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-03-25T00:38:56.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T00:38:57.034 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' already exists 2024-03-25T00:38:57.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:38:58.052 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-03-25T00:38:58.052 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:38:58.052 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:38:58.053 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:38:58.053 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:38:58.053 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:38:58.053 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:38:58.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-03-25T00:38:58.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-mon.a.asok log flush 2024-03-25T00:38:58.153 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:38:58.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-03-25T00:38:58.169 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:38:55.403+0000 7f0703ee7c80 0 load: jerasure load: lrc 2024-03-25T00:38:58.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2024-03-25T00:38:58.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2024-03-25T00:38:58.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:481: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2024-03-25T00:38:58.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:38:58.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:38:58.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:485: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2024-03-25T00:38:58.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-03-25T00:38:58.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:38:58.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:38:58.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:38:58.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a488d7a7-1d9a-4467-864f-0c1a3631d111 2024-03-25T00:38:58.180 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 a488d7a7-1d9a-4467-864f-0c1a3631d111 2024-03-25T00:38:58.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a488d7a7-1d9a-4467-864f-0c1a3631d111' 2024-03-25T00:38:58.180 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:38:58.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCixwBm8ovfCxAAaze9h8TXUETbGljXLhIOmA== 2024-03-25T00:38:58.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCixwBm8ovfCxAAaze9h8TXUETbGljXLhIOmA=="}' 2024-03-25T00:38:58.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a488d7a7-1d9a-4467-864f-0c1a3631d111 -i td/test-erasure-eio/0/new.json 2024-03-25T00:38:58.486 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:38:58.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-03-25T00:38:58.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCixwBm8ovfCxAAaze9h8TXUETbGljXLhIOmA== --osd-uuid a488d7a7-1d9a-4467-864f-0c1a3631d111 2024-03-25T00:38:58.540 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:58.539+0000 7f59d27537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:38:58.553 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:58.551+0000 7f59d27537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:38:58.555 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:58.555+0000 7f59d27537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:38:58.555 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:58.555+0000 7f59d27537c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:38:58.555 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:58.555+0000 7f59d27537c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:38:58.555 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:58.555+0000 7f59d27537c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:38:58.594 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:38:58.591+0000 7f59d27537c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-03-25T00:39:04.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-03-25T00:39:04.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:39:04.758 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:39:04.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:39:04.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:39:05.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:39:05.324 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:39:05.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:39:05.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:39:05.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:39:05.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:39:05.359 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:05.359+0000 7f8d6d75a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:39:05.383 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:05.383+0000 7f8d6d75a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:39:05.384 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:05.383+0000 7f8d6d75a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:39:05.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:39:05.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:39:05.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:39:05.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:39:05.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:39:05.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:05.761 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:39:05.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:39:05.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:05.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:05.972 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:05.971+0000 7f8d6d75a7c0 -1 Falling back to public interface 2024-03-25T00:39:06.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:07.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:07.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:07.212 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:39:07.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:39:07.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:07.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:07.641 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:08.491 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:08.491+0000 7f8d6d75a7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:39:08.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:08.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:08.644 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:39:08.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:39:08.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:08.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:09.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:10.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:10.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:10.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:39:10.154 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:39:10.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:10.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:10.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:11.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:11.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:11.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:39:11.583 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:39:11.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:11.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:12.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:13.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:13.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:13.022 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:39:13.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:39:13.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:13.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:13.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:14.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:14.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:14.593 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:39:14.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:39:14.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:14.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:15.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:16.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:16.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:16.032 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:39:16.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:39:16.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:16.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:16.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:17.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:17.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:17.574 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:39:17.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:39:17.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:17.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:17.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:18.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:18.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:18.994 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:39:18.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:39:18.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:18.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:19.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:20.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:20.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:20.427 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:39:20.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:39:20.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:20.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:21.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:22.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:22.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:22.027 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:39:22.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:39:22.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:22.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:22.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:23.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:23.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:23.917 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:39:23.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:39:23.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:23.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:24.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:25.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:25.765 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:25.766 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:39:25.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:39:25.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:25.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:26.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:27.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:27.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:27.569 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:39:27.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:39:27.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:27.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:28.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:29.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:29.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:29.537 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:39:29.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:39:29.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:29.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:30.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:31.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:31.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:31.317 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:39:31.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:39:31.317 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:31.318 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:31.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:32.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:32.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:32.804 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:39:32.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:39:32.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:32.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:39:33.300 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4188252110,v1:127.0.0.1:6803/4188252110] [v2:127.0.0.1:6804/4188252110,v1:127.0.0.1:6805/4188252110] exists,up a488d7a7-1d9a-4467-864f-0c1a3631d111 2024-03-25T00:39:33.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:39:33.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:39:33.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:39:33.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:39:33.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:39:33.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:39:33.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:39:33.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:39:33.303 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:39:33.303 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:39:33.303 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:39:33.304 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:39:33.304 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:39:33.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:39:33.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:39:33.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:39:33.306 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:39:33.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d68e2fa9-ceb9-415b-8309-5aac40e80d17 2024-03-25T00:39:33.306 INFO:tasks.workunit.client.0.smithi080.stdout:add osd1 d68e2fa9-ceb9-415b-8309-5aac40e80d17 2024-03-25T00:39:33.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d68e2fa9-ceb9-415b-8309-5aac40e80d17' 2024-03-25T00:39:33.307 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:39:33.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDFxwBmoIrOExAA2cn5nwfpFg7epSBt40ZC8Q== 2024-03-25T00:39:33.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDFxwBmoIrOExAA2cn5nwfpFg7epSBt40ZC8Q=="}' 2024-03-25T00:39:33.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d68e2fa9-ceb9-415b-8309-5aac40e80d17 -i td/test-erasure-eio/1/new.json 2024-03-25T00:39:33.802 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:39:33.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-03-25T00:39:33.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFxwBmoIrOExAA2cn5nwfpFg7epSBt40ZC8Q== --osd-uuid d68e2fa9-ceb9-415b-8309-5aac40e80d17 2024-03-25T00:39:33.850 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:33.846+0000 7f05caa607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:39:33.872 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:33.870+0000 7f05caa607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:39:33.873 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:33.870+0000 7f05caa607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:39:33.873 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:33.870+0000 7f05caa607c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:39:33.874 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:33.870+0000 7f05caa607c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:39:33.874 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:33.870+0000 7f05caa607c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:39:33.911 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:33.910+0000 7f05caa607c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-03-25T00:39:40.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-03-25T00:39:40.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:39:40.042 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd1 key to auth repository 2024-03-25T00:39:40.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T00:39:40.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:39:40.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T00:39:40.595 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.1 2024-03-25T00:39:40.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:39:40.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:39:40.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:39:40.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:39:40.627 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:40.626+0000 7fc2d7e217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:39:40.646 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:40.642+0000 7fc2d7e217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:39:40.647 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:40.646+0000 7fc2d7e217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:39:41.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T00:39:41.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:39:41.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:39:41.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:39:41.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:39:41.022 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:41.023 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:39:41.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:39:41.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:41.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:41.220 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:41.218+0000 7fc2d7e217c0 -1 Falling back to public interface 2024-03-25T00:39:41.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:42.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:42.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:42.441 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:39:42.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:39:42.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:42.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:42.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:43.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:43.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:43.846 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:39:43.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:39:43.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:43.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:43.959 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:39:43.958+0000 7fc2d7e217c0 -1 osd.1 0 log_to_monitors true 2024-03-25T00:39:44.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:45.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:45.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:45.302 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:39:45.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:39:45.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:45.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:45.747 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:46.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:46.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:46.748 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:39:46.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:39:46.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:46.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:47.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:48.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:48.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:48.258 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:39:48.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:39:48.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:48.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:48.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:49.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:49.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:49.688 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:39:49.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:39:49.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:49.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:50.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:51.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:51.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:51.229 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:39:51.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:39:51.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:51.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:51.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:52.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:52.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:52.773 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:39:52.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:39:52.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:52.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:53.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:54.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:54.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:54.247 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:39:54.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:39:54.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:54.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:54.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:55.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:55.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:55.674 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:39:55.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:39:55.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:55.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:56.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:57.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:57.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:57.251 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:39:57.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:39:57.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:57.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:39:58.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:39:59.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:39:59.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:39:59.178 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:39:59.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:39:59.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:39:59.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:40:00.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:01.102 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:01.102 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:01.102 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:40:01.102 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:40:01.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:01.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:40:01.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:02.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:02.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:02.871 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:40:02.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:40:02.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:02.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:40:03.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:04.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:04.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:04.582 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:40:04.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:40:04.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:04.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:40:05.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:06.403 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:06.403 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:06.403 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:40:06.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:40:06.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:06.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:40:07.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:08.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:08.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:08.206 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:40:08.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:40:08.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:08.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:40:08.628 INFO:tasks.workunit.client.0.smithi080.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/129440946,v1:127.0.0.1:6811/129440946] [v2:127.0.0.1:6812/129440946,v1:127.0.0.1:6813/129440946] exists,up d68e2fa9-ceb9-415b-8309-5aac40e80d17 2024-03-25T00:40:08.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:40:08.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:40:08.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:40:08.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:40:08.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:40:08.632 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:40:08.632 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:40:08.632 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:40:08.632 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:40:08.632 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:40:08.633 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:40:08.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:40:08.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:40:08.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:40:08.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:40:08.635 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:40:08.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=81808f7c-9d3e-406b-807a-be19e68b7b71 2024-03-25T00:40:08.635 INFO:tasks.workunit.client.0.smithi080.stdout:add osd2 81808f7c-9d3e-406b-807a-be19e68b7b71 2024-03-25T00:40:08.636 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 81808f7c-9d3e-406b-807a-be19e68b7b71' 2024-03-25T00:40:08.636 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:40:08.655 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDoxwBmFPUdJxAAwGRM/USQb+s3QeZ3Z9Rr5Q== 2024-03-25T00:40:08.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDoxwBmFPUdJxAAwGRM/USQb+s3QeZ3Z9Rr5Q=="}' 2024-03-25T00:40:08.656 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 81808f7c-9d3e-406b-807a-be19e68b7b71 -i td/test-erasure-eio/2/new.json 2024-03-25T00:40:09.598 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:40:09.617 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-03-25T00:40:09.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDoxwBmFPUdJxAAwGRM/USQb+s3QeZ3Z9Rr5Q== --osd-uuid 81808f7c-9d3e-406b-807a-be19e68b7b71 2024-03-25T00:40:09.654 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:09.650+0000 7f895b91b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:09.657 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:09.654+0000 7f895b91b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:09.658 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:09.654+0000 7f895b91b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:09.658 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:09.654+0000 7f895b91b7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:40:09.658 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:09.654+0000 7f895b91b7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:40:09.658 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:09.654+0000 7f895b91b7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:40:09.740 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:09.738+0000 7f895b91b7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-03-25T00:40:16.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-03-25T00:40:16.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:40:16.060 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd2 key to auth repository 2024-03-25T00:40:16.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T00:40:16.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:40:16.606 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:40:16.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T00:40:16.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:40:16.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:40:16.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:40:16.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:40:16.642 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:16.638+0000 7f0baf7177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:16.665 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:16.662+0000 7f0baf7177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:16.666 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:16.666+0000 7f0baf7177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:17.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T00:40:17.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:40:17.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:40:17.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:40:17.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:40:17.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:17.079 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:40:17.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:40:17.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:17.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:17.224 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:17.222+0000 7f0baf7177c0 -1 Falling back to public interface 2024-03-25T00:40:17.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:18.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:18.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:18.497 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:40:18.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:40:18.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:18.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:18.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:19.789 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:19.785+0000 7f0baf7177c0 -1 osd.2 0 log_to_monitors true 2024-03-25T00:40:19.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:19.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:19.903 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:40:19.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:40:19.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:19.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:20.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:21.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:21.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:21.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:40:21.489 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:40:21.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:21.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:22.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:23.064 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:23.064 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:23.064 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:40:23.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:40:23.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:23.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:23.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:24.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:24.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:24.697 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:40:24.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:40:24.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:24.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:25.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:26.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:26.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:26.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:40:26.119 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:40:26.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:26.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:26.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:27.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:27.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:27.592 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:40:27.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:40:27.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:27.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:28.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:29.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:29.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:29.143 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:40:29.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:40:29.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:29.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:29.661 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:30.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:30.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:30.664 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:40:30.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:40:30.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:30.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:31.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:32.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:32.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:32.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:40:32.109 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:40:32.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:32.109 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:32.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:33.603 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:33.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:33.604 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:40:33.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:40:33.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:33.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:34.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:35.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:35.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:35.751 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:40:35.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:40:35.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:35.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:37.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:38.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:38.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:38.186 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:40:38.187 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:40:38.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:38.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:39.146 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:40.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:40.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:40.148 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:40:40.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:40:40.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:40.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:40.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:41.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:41.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:41.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:40:41.626 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:40:41.627 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:41.627 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:42.477 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:43.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:43.479 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:43.480 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:40:43.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:40:43.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:43.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:40:43.967 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1551879286,v1:127.0.0.1:6819/1551879286] [v2:127.0.0.1:6820/1551879286,v1:127.0.0.1:6821/1551879286] exists,up 81808f7c-9d3e-406b-807a-be19e68b7b71 2024-03-25T00:40:43.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:40:43.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:40:43.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:40:43.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:40:43.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:40:43.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:40:43.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:40:43.972 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:40:43.972 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:40:43.972 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:40:43.972 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:40:43.972 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:40:43.972 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:40:43.972 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:40:43.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:40:43.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:40:43.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:40:43.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:40:43.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:40:43.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:40:43.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:40:43.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:40:43.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:40:43.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:40:43.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:40:43.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:40:43.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:40:43.974 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:40:43.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b3bc907f-e452-4cc9-a6a7-831eb5445214 2024-03-25T00:40:43.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b3bc907f-e452-4cc9-a6a7-831eb5445214' 2024-03-25T00:40:43.975 INFO:tasks.workunit.client.0.smithi080.stdout:add osd3 b3bc907f-e452-4cc9-a6a7-831eb5445214 2024-03-25T00:40:43.975 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:40:43.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALyABmJc44OxAAFRVebSPSk6faRELY/jFRNg== 2024-03-25T00:40:43.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALyABmJc44OxAAFRVebSPSk6faRELY/jFRNg=="}' 2024-03-25T00:40:43.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b3bc907f-e452-4cc9-a6a7-831eb5445214 -i td/test-erasure-eio/3/new.json 2024-03-25T00:40:44.476 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:40:44.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-03-25T00:40:44.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALyABmJc44OxAAFRVebSPSk6faRELY/jFRNg== --osd-uuid b3bc907f-e452-4cc9-a6a7-831eb5445214 2024-03-25T00:40:44.524 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:44.521+0000 7efe021ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:44.679 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:44.677+0000 7efe021ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:44.681 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:44.677+0000 7efe021ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:44.681 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:44.677+0000 7efe021ce7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:40:44.681 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:44.677+0000 7efe021ce7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:40:44.681 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:44.677+0000 7efe021ce7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:40:44.778 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:44.773+0000 7efe021ce7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-03-25T00:40:52.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-03-25T00:40:52.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:40:52.137 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd3 key to auth repository 2024-03-25T00:40:52.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T00:40:52.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:40:52.696 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.3 2024-03-25T00:40:52.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T00:40:52.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:40:52.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:40:52.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:40:52.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:40:52.733 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:52.729+0000 7f4bf53cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:52.756 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:52.753+0000 7f4bf53cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:52.757 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:52.753+0000 7f4bf53cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:40:53.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T00:40:53.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:40:53.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:40:53.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:40:53.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:40:53.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:53.126 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:40:53.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:40:53.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:40:53.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:53.333 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:53.329+0000 7f4bf53cf7c0 -1 Falling back to public interface 2024-03-25T00:40:53.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:54.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:54.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:54.558 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:40:54.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:40:54.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:54.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:40:55.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:56.035 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:40:56.029+0000 7f4bf53cf7c0 -1 osd.3 0 log_to_monitors true 2024-03-25T00:40:56.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:56.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:56.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:40:56.068 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:40:56.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:56.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:40:56.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:57.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:57.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:57.787 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:40:57.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:40:57.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:57.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:40:58.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:40:59.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:40:59.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:40:59.381 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:40:59.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:40:59.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:40:59.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:40:59.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:00.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:00.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:00.913 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:41:00.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:41:00.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:00.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:01.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:02.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:02.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:02.344 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:41:02.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:41:02.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:02.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:02.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:03.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:03.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:03.911 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:41:03.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:41:03.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:03.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:04.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:05.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:05.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:05.401 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:41:05.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:41:05.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:05.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:05.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:06.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:06.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:06.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:41:06.932 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:41:06.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:06.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:07.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:08.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:08.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:08.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:41:08.390 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:41:08.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:08.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:08.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:09.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:09.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:09.912 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:41:09.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:41:09.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:09.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:10.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:11.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:11.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:11.804 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:41:11.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:41:11.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:11.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:12.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:13.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:13.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:13.708 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:41:13.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:41:13.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:13.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:14.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:15.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:15.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:15.484 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:41:15.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:41:15.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:15.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:15.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:16.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:16.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:16.928 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:41:16.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:41:16.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:16.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:17.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:18.836 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:18.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:18.837 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:41:18.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:41:18.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:18.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:19.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:20.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:20.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:20.713 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:41:20.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:41:20.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:20.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:41:21.205 INFO:tasks.workunit.client.0.smithi080.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/769575069,v1:127.0.0.1:6827/769575069] [v2:127.0.0.1:6828/769575069,v1:127.0.0.1:6829/769575069] exists,up b3bc907f-e452-4cc9-a6a7-831eb5445214 2024-03-25T00:41:21.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:41:21.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:41:21.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:41:21.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:41:21.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-03-25T00:41:21.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:41:21.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:41:21.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-25T00:41:21.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:41:21.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-03-25T00:41:21.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:41:21.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:41:21.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:41:21.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:41:21.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-03-25T00:41:21.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-03-25T00:41:21.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:41:21.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:41:21.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:41:21.211 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:41:21.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:41:21.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:41:21.212 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:41:21.212 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:41:21.212 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:41:21.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:41:21.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:41:21.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:41:21.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:41:21.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:41:21.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:41:21.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:41:21.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:41:21.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:41:21.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:41:21.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:41:21.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:41:21.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:41:21.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-03-25T00:41:21.213 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:41:21.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e950577b-46af-46ef-a0f9-264dd851adc2 2024-03-25T00:41:21.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 e950577b-46af-46ef-a0f9-264dd851adc2' 2024-03-25T00:41:21.214 INFO:tasks.workunit.client.0.smithi080.stdout:add osd4 e950577b-46af-46ef-a0f9-264dd851adc2 2024-03-25T00:41:21.214 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:41:21.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxyABmwaciDhAAqnqURGHZPFm6ZSgaXeHSbQ== 2024-03-25T00:41:21.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxyABmwaciDhAAqnqURGHZPFm6ZSgaXeHSbQ=="}' 2024-03-25T00:41:21.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e950577b-46af-46ef-a0f9-264dd851adc2 -i td/test-erasure-eio/4/new.json 2024-03-25T00:41:21.940 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:41:21.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-03-25T00:41:21.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxyABmwaciDhAAqnqURGHZPFm6ZSgaXeHSbQ== --osd-uuid e950577b-46af-46ef-a0f9-264dd851adc2 2024-03-25T00:41:21.996 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:21.996+0000 7ff2322977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:41:21.998 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:21.996+0000 7ff2322977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:41:21.999 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:22.000+0000 7ff2322977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:41:21.999 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:22.000+0000 7ff2322977c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:41:22.000 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:22.000+0000 7ff2322977c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:41:22.000 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:22.000+0000 7ff2322977c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:41:22.192 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:22.192+0000 7ff2322977c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-03-25T00:41:29.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-03-25T00:41:29.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:41:29.118 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd4 key to auth repository 2024-03-25T00:41:29.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-25T00:41:29.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:41:29.673 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.4 2024-03-25T00:41:29.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-25T00:41:29.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:41:29.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:41:29.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:41:29.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:41:29.711 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:29.712+0000 7f9288aab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:41:29.738 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:29.736+0000 7f9288aab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:41:29.739 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:29.740+0000 7f9288aab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:41:30.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-25T00:41:30.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:41:30.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-25T00:41:30.127 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:41:30.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:41:30.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:41:30.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:30.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:41:30.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:30.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:30.312 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:30.312+0000 7f9288aab7c0 -1 Falling back to public interface 2024-03-25T00:41:30.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:31.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:31.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:31.535 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:41:31.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:41:31.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:31.535 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:31.967 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:32.821 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:41:32.820+0000 7f9288aab7c0 -1 osd.4 0 log_to_monitors true 2024-03-25T00:41:32.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:32.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:32.969 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:41:32.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:41:32.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:32.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:33.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:34.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:34.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:34.685 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:41:34.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:41:34.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:34.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:35.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:36.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:36.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:36.323 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:41:36.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:41:36.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:36.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:36.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:37.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:37.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:37.949 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:41:37.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:41:37.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:37.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:38.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:39.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:39.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:39.463 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:41:39.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:41:39.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:39.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:40.116 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:41.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:41.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:41.119 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:41:41.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:41:41.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:41.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:41.747 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:42.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:42.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:42.749 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:41:42.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:41:42.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:42.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:43.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:44.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:44.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:44.214 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:41:44.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:41:44.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:44.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:44.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:45.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:45.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:45.821 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:41:45.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:41:45.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:45.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:46.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:47.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:47.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:47.337 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:41:47.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:41:47.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:47.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:48.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:49.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:49.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:49.551 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:41:49.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:41:49.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:49.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:50.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:51.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:51.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:51.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:41:51.510 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:41:51.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:51.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:51.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:52.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:52.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:52.955 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:41:52.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:41:52.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:52.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:53.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:54.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:54.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:54.701 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:41:54.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:41:54.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:54.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:55.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:56.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:56.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:56.783 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:41:56.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:41:56.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:56.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:57.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:41:58.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:41:58.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:41:58.752 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:41:58.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:41:58.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:41:58.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:41:59.186 INFO:tasks.workunit.client.0.smithi080.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/93394429,v1:127.0.0.1:6835/93394429] [v2:127.0.0.1:6836/93394429,v1:127.0.0.1:6837/93394429] exists,up e950577b-46af-46ef-a0f9-264dd851adc2 2024-03-25T00:41:59.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:41:59.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:41:59.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:41:59.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:41:59.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-03-25T00:41:59.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:41:59.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:41:59.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-25T00:41:59.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:41:59.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:41:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:41:59.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:41:59.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-03-25T00:41:59.193 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:41:59.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=38f793f6-99af-4043-8399-47efe382d8c1 2024-03-25T00:41:59.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 38f793f6-99af-4043-8399-47efe382d8c1' 2024-03-25T00:41:59.194 INFO:tasks.workunit.client.0.smithi080.stdout:add osd5 38f793f6-99af-4043-8399-47efe382d8c1 2024-03-25T00:41:59.194 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:41:59.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXyABmho/ODBAAmK6CwCltDGWC+WyhCcyphg== 2024-03-25T00:41:59.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXyABmho/ODBAAmK6CwCltDGWC+WyhCcyphg=="}' 2024-03-25T00:41:59.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 38f793f6-99af-4043-8399-47efe382d8c1 -i td/test-erasure-eio/5/new.json 2024-03-25T00:41:59.963 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:41:59.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-03-25T00:41:59.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXyABmho/ODBAAmK6CwCltDGWC+WyhCcyphg== --osd-uuid 38f793f6-99af-4043-8399-47efe382d8c1 2024-03-25T00:42:00.024 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:00.023+0000 7feff2c587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:00.070 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:00.067+0000 7feff2c587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:00.071 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:00.071+0000 7feff2c587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:00.072 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:00.071+0000 7feff2c587c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:42:00.072 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:00.071+0000 7feff2c587c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:42:00.072 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:00.071+0000 7feff2c587c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:42:00.211 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:00.211+0000 7feff2c587c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-03-25T00:42:06.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-03-25T00:42:06.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:42:06.902 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd5 key to auth repository 2024-03-25T00:42:06.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-25T00:42:06.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:42:07.496 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.5 2024-03-25T00:42:07.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-25T00:42:07.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:42:07.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:42:07.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:42:07.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:42:07.532 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:07.531+0000 7f11b719d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:07.546 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:07.543+0000 7f11b719d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:07.547 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:07.547+0000 7f11b719d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:07.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-25T00:42:07.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:42:07.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T00:42:07.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:42:07.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:42:07.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:07.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:42:07.952 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:42:07.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:07.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:08.136 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:08.135+0000 7f11b719d7c0 -1 Falling back to public interface 2024-03-25T00:42:08.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:09.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:09.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:09.362 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:42:09.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:42:09.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:09.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:09.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:10.635 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:10.635+0000 7f11b719d7c0 -1 osd.5 0 log_to_monitors true 2024-03-25T00:42:10.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:10.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:10.769 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:42:10.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:42:10.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:10.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:11.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:12.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:12.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:12.375 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:42:12.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:42:12.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:12.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:12.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:14.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:14.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:14.002 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:42:14.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:42:14.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:14.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:14.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:15.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:15.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:15.556 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:42:15.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:42:15.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:15.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:16.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:17.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:17.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:17.091 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:42:17.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:42:17.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:17.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:17.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:18.540 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:18.541 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:18.541 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:42:18.541 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:42:18.542 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:18.542 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:19.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:20.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:20.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:20.067 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:42:20.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:42:20.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:20.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:20.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:21.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:21.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:21.586 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:42:21.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:42:21.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:21.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:22.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:23.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:23.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:23.128 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:42:23.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:42:23.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:23.128 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:23.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:24.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:24.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:24.568 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:42:24.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:42:24.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:24.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:25.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:26.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:26.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:26.092 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:42:26.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:42:26.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:26.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:27.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:28.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:28.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:28.297 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:42:28.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:42:28.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:28.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:29.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:30.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:30.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:30.028 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:42:30.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:42:30.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:30.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:30.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:31.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:31.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:31.888 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:42:31.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:42:31.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:31.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:32.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:33.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:33.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:33.744 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:42:33.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:42:33.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:33.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:34.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:35.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:35.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:35.851 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:42:35.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:42:35.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:35.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:42:36.305 INFO:tasks.workunit.client.0.smithi080.stdout:osd.5 up in weight 1 up_from 47 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3305244632,v1:127.0.0.1:6843/3305244632] [v2:127.0.0.1:6844/3305244632,v1:127.0.0.1:6845/3305244632] exists,up 38f793f6-99af-4043-8399-47efe382d8c1 2024-03-25T00:42:36.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:42:36.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:42:36.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:42:36.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:42:36.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-03-25T00:42:36.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:42:36.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:42:36.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:42:36.310 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:42:36.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:42:36.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:42:36.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:42:36.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-03-25T00:42:36.312 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:42:36.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0bb81fc7-ef02-4e36-a4ac-dfebb24a0908 2024-03-25T00:42:36.313 INFO:tasks.workunit.client.0.smithi080.stdout:add osd6 0bb81fc7-ef02-4e36-a4ac-dfebb24a0908 2024-03-25T00:42:36.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 0bb81fc7-ef02-4e36-a4ac-dfebb24a0908' 2024-03-25T00:42:36.313 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:42:36.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8yABmrpA8FBAA1ygMBlsd2Ix5ukh/sD8KeQ== 2024-03-25T00:42:36.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8yABmrpA8FBAA1ygMBlsd2Ix5ukh/sD8KeQ=="}' 2024-03-25T00:42:36.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0bb81fc7-ef02-4e36-a4ac-dfebb24a0908 -i td/test-erasure-eio/6/new.json 2024-03-25T00:42:37.196 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:42:37.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-03-25T00:42:37.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8yABmrpA8FBAA1ygMBlsd2Ix5ukh/sD8KeQ== --osd-uuid 0bb81fc7-ef02-4e36-a4ac-dfebb24a0908 2024-03-25T00:42:37.257 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:37.254+0000 7f7c253b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:37.353 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:37.350+0000 7f7c253b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:37.355 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:37.354+0000 7f7c253b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:37.356 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:37.354+0000 7f7c253b17c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:42:37.356 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:37.354+0000 7f7c253b17c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:42:37.356 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:37.354+0000 7f7c253b17c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:42:37.515 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:37.514+0000 7f7c253b17c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-03-25T00:42:44.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-03-25T00:42:44.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:42:44.153 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd6 key to auth repository 2024-03-25T00:42:44.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-25T00:42:44.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:42:44.736 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.6 2024-03-25T00:42:44.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-25T00:42:44.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:42:44.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:42:44.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:42:44.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:42:44.773 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:44.770+0000 7f9f563687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:44.781 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:44.778+0000 7f9f563687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:44.782 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:44.778+0000 7f9f563687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:42:45.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-25T00:42:45.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:42:45.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-25T00:42:45.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:42:45.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:42:45.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:45.158 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:42:45.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:42:45.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:45.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:42:45.376 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:45.374+0000 7f9f563687c0 -1 Falling back to public interface 2024-03-25T00:42:45.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:46.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:46.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:46.593 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:42:46.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:42:46.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:46.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:42:47.007 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:48.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:48.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:48.009 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:42:48.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:42:48.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:48.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:42:48.033 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:42:48.030+0000 7f9f563687c0 -1 osd.6 0 log_to_monitors true 2024-03-25T00:42:48.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:49.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:49.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:49.621 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:42:49.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:42:49.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:49.622 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:42:50.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:51.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:51.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:51.397 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:42:51.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:42:51.398 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:51.398 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:42:52.074 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:53.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:53.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:53.076 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:42:53.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:42:53.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:53.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:42:53.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:54.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:54.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:54.500 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:42:54.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:42:54.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:54.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:42:55.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:56.064 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:56.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:56.065 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:42:56.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:42:56.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:56.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:42:56.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:57.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:57.651 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:57.651 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:42:57.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:42:57.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:57.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:42:58.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:42:59.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:42:59.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:42:59.283 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:42:59.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:42:59.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:42:59.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:42:59.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:00.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:00.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:00.962 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:43:00.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:43:00.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:00.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:43:01.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:02.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:02.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:02.397 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:43:02.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:43:02.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:02.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:43:03.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:04.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:04.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:04.245 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:43:04.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:43:04.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:04.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:43:05.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:06.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:06.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:06.127 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:43:06.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:43:06.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:06.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:43:07.112 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:08.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:08.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:08.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:43:08.114 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:43:08.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:08.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:43:09.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:10.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:10.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:10.028 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:43:10.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:43:10.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:10.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:43:10.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:11.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:11.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:11.686 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:43:11.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:43:11.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:11.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:43:12.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:13.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:13.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:13.526 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:43:13.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:43:13.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:13.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:43:14.220 INFO:tasks.workunit.client.0.smithi080.stdout:osd.6 up in weight 1 up_from 57 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2362901246,v1:127.0.0.1:6851/2362901246] [v2:127.0.0.1:6852/2362901246,v1:127.0.0.1:6853/2362901246] exists,up 0bb81fc7-ef02-4e36-a4ac-dfebb24a0908 2024-03-25T00:43:14.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:43:14.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:43:14.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:43:14.222 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-03-25T00:43:14.222 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:43:14.222 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:43:14.223 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:43:14.223 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:14.223 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:14.223 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.0.asok 2024-03-25T00:43:14.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-03-25T00:43:14.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.0.asok log flush 2024-03-25T00:43:14.337 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:43:14.350 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-03-25T00:43:14.357 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:39:05.979+0000 7f8d6d75a7c0 0 load: jerasure load: lrc 2024-03-25T00:43:14.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:43:14.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:488: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2024-03-25T00:43:14.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2024-03-25T00:43:14.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:43:14.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-03-25T00:43:14.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-03-25T00:43:14.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-03-25T00:43:14.359 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-03-25T00:43:14.359 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-03-25T00:43:14.359 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-03-25T00:43:15.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-03-25T00:43:15.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-03-25T00:43:16.397 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' already exists 2024-03-25T00:43:16.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:43:17.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-03-25T00:43:17.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:43:17.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:43:17.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:43:17.418 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:43:17.418 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:43:17.419 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:43:17.419 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:43:17.419 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:43:17.419 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:43:17.541 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:43:17.542 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:43:17.542 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:43:17.542 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:43:17.542 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:43:17.542 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:43:18.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:43:18.160 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:43:18.160 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:43:18.160 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:43:18.160 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:43:18.161 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:43:18.161 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:43:18.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:43:18.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:18.161 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:43:18.318 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803826 2024-03-25T00:43:18.318 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803826 2024-03-25T00:43:18.318 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826' 2024-03-25T00:43:18.319 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:18.319 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:43:18.470 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607599 2024-03-25T00:43:18.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607599 2024-03-25T00:43:18.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599' 2024-03-25T00:43:18.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:18.471 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:43:18.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411369 2024-03-25T00:43:18.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411369 2024-03-25T00:43:18.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599 2-77309411369' 2024-03-25T00:43:18.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:18.624 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:43:18.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117024 2024-03-25T00:43:18.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117024 2024-03-25T00:43:18.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599 2-77309411369 3-115964117024' 2024-03-25T00:43:18.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:18.773 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:43:18.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757271 2024-03-25T00:43:18.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757271 2024-03-25T00:43:18.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599 2-77309411369 3-115964117024 4-163208757271' 2024-03-25T00:43:18.925 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:18.925 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:43:19.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462928 2024-03-25T00:43:19.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462928 2024-03-25T00:43:19.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599 2-77309411369 3-115964117024 4-163208757271 5-201863462928' 2024-03-25T00:43:19.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:19.079 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:43:19.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135877 2024-03-25T00:43:19.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135877 2024-03-25T00:43:19.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607599 2-77309411369 3-115964117024 4-163208757271 5-201863462928 6-244813135877' 2024-03-25T00:43:19.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:19.236 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803826 2024-03-25T00:43:19.236 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:19.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:43:19.237 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803826 2024-03-25T00:43:19.237 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:19.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803826 2024-03-25T00:43:19.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-03-25T00:43:19.239 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803826 2024-03-25T00:43:19.239 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:43:19.627 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803826 -lt 25769803826 2024-03-25T00:43:19.627 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:19.628 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607599 2024-03-25T00:43:19.628 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:19.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:43:19.630 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607599 2024-03-25T00:43:19.630 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:19.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607599 2024-03-25T00:43:19.631 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 51539607599 2024-03-25T00:43:19.632 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607599' 2024-03-25T00:43:19.632 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:43:20.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607600 -lt 51539607599 2024-03-25T00:43:20.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:20.072 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:20.072 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411369 2024-03-25T00:43:20.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:43:20.073 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411369 2024-03-25T00:43:20.075 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:20.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411369 2024-03-25T00:43:20.075 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 77309411369 2024-03-25T00:43:20.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411369' 2024-03-25T00:43:20.076 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:43:20.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411370 -lt 77309411369 2024-03-25T00:43:20.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:20.630 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117024 2024-03-25T00:43:20.630 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:20.632 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:43:20.633 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117024 2024-03-25T00:43:20.633 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:20.634 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117024 2024-03-25T00:43:20.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117024' 2024-03-25T00:43:20.635 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 115964117024 2024-03-25T00:43:20.635 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:43:21.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117024 -lt 115964117024 2024-03-25T00:43:21.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:21.106 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757271 2024-03-25T00:43:21.106 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:21.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:43:21.109 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757271 2024-03-25T00:43:21.109 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:21.110 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 163208757271 2024-03-25T00:43:21.111 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757271 2024-03-25T00:43:21.111 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757271' 2024-03-25T00:43:21.111 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:43:21.541 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757271 -lt 163208757271 2024-03-25T00:43:21.542 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:21.542 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-201863462928 2024-03-25T00:43:21.543 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:21.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:43:21.545 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:21.545 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-201863462928 2024-03-25T00:43:21.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462928 2024-03-25T00:43:21.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 201863462928' 2024-03-25T00:43:21.546 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 201863462928 2024-03-25T00:43:21.546 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:43:21.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462929 -lt 201863462928 2024-03-25T00:43:21.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:21.975 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-244813135877 2024-03-25T00:43:21.975 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:21.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:43:21.977 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-244813135877 2024-03-25T00:43:21.977 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:21.978 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135877 2024-03-25T00:43:21.978 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 244813135877' 2024-03-25T00:43:21.978 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 244813135877 2024-03-25T00:43:21.978 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:43:22.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135877 -lt 244813135877 2024-03-25T00:43:22.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:43:22.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:43:22.447 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:43:22.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:43:22.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:43:22.983 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:43:22.983 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:43:22.983 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:43:22.984 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:43:22.984 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:43:22.984 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:43:23.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:43:23.393 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:43:23.393 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:43:23.393 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:43:23.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:43:23.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:43:23.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:43:23.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:43:23.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-03-25T00:43:24.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2024-03-25T00:43:24.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:43:24.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:43:24.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-03-25T00:43:24.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:24.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:43:24.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:24.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:43:24.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:24.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:43:24.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:24.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:43:24.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-03-25T00:43:24.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:43:24.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-03-25T00:43:24.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:496: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-03-25T00:43:24.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-03-25T00:43:24.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-25T00:43:24.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-25T00:43:24.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-25T00:43:24.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-03-25T00:43:24.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-25T00:43:24.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-03-25T00:43:24.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-25T00:43:24.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-03-25T00:43:24.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-25T00:43:24.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-25T00:43:24.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-25T00:43:24.743 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-03-25T00:43:24.743 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:43:24.743 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-03-25T00:43:24.743 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:43:24.743 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:43:25.159 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:43:25.159 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:43:25.159 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:43:25.160 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:43:25.160 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:43:25.160 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-03-25T00:43:25.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-03-25T00:43:25.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-25T00:43:25.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-03-25T00:43:25.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-03-25T00:43:25.161 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-03-25T00:43:25.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-25T00:43:25.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-03-25T00:43:25.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T00:43:25.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-03-25T00:43:25.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-25T00:43:25.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-25T00:43:25.163 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T00:43:25.163 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-03-25T00:43:25.164 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-03-25T00:43:25.164 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-03-25T00:43:25.164 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:43:25.164 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:25.164 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:25.164 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.3.asok 2024-03-25T00:43:25.164 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81310/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-03-25T00:43:25.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T00:43:25.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-25T00:43:25.273 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-03-25T00:43:25.273 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-03-25T00:43:25.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-25T00:43:25.273 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-03-25T00:43:25.274 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:43:25.274 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:25.274 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:25.274 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.3.asok 2024-03-25T00:43:25.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-25T00:43:25.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-03-25T00:43:25.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:497: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2024-03-25T00:43:25.894 INFO:tasks.workunit.client.0.smithi080.stderr:osd.0 is already out. 2024-03-25T00:43:25.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2024-03-25T00:43:25.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:43:25.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:43:25.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:43:25.910 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:43:25.910 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:43:25.910 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:43:25.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:43:25.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:43:25.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:43:26.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:43:26.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:43:26.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:43:26.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:43:26.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:43:26.010 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:43:26.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:43:26.502 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:43:26.502 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:43:26.502 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:43:26.502 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:43:26.502 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:43:26.502 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:43:26.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:43:26.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:26.503 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:43:26.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803829 2024-03-25T00:43:26.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803829 2024-03-25T00:43:26.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829' 2024-03-25T00:43:26.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:26.671 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:43:26.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607603 2024-03-25T00:43:26.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607603 2024-03-25T00:43:26.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-51539607603' 2024-03-25T00:43:26.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:26.907 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:43:27.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411373 2024-03-25T00:43:27.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411373 2024-03-25T00:43:27.059 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-51539607603 2-77309411373' 2024-03-25T00:43:27.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:27.060 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:43:27.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117028 2024-03-25T00:43:27.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117028 2024-03-25T00:43:27.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-51539607603 2-77309411373 3-115964117028' 2024-03-25T00:43:27.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:27.202 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:43:27.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757274 2024-03-25T00:43:27.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757274 2024-03-25T00:43:27.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-51539607603 2-77309411373 3-115964117028 4-163208757274' 2024-03-25T00:43:27.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:27.349 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:43:27.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462932 2024-03-25T00:43:27.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462932 2024-03-25T00:43:27.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-51539607603 2-77309411373 3-115964117028 4-163208757274 5-201863462932' 2024-03-25T00:43:27.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:43:27.511 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:43:27.669 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135880 2024-03-25T00:43:27.669 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135880 2024-03-25T00:43:27.669 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-51539607603 2-77309411373 3-115964117028 4-163208757274 5-201863462932 6-244813135880' 2024-03-25T00:43:27.669 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:27.670 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803829 2024-03-25T00:43:27.670 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:27.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:43:27.671 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803829 2024-03-25T00:43:27.671 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:27.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803829 2024-03-25T00:43:27.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-03-25T00:43:27.672 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803829 2024-03-25T00:43:27.672 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:43:28.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803829 -lt 25769803829 2024-03-25T00:43:28.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:28.287 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:28.287 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607603 2024-03-25T00:43:28.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:43:28.289 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607603 2024-03-25T00:43:28.289 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:28.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607603 2024-03-25T00:43:28.290 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 51539607603 2024-03-25T00:43:28.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607603' 2024-03-25T00:43:28.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:43:28.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607603 -lt 51539607603 2024-03-25T00:43:28.693 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:28.694 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:28.694 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411373 2024-03-25T00:43:28.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:43:28.695 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411373 2024-03-25T00:43:28.696 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:28.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411373 2024-03-25T00:43:28.697 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 77309411373 2024-03-25T00:43:28.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411373' 2024-03-25T00:43:28.697 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:43:29.122 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411373 -lt 77309411373 2024-03-25T00:43:29.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:29.123 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117028 2024-03-25T00:43:29.123 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:29.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:43:29.125 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117028 2024-03-25T00:43:29.125 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:29.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117028 2024-03-25T00:43:29.127 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 115964117028 2024-03-25T00:43:29.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117028' 2024-03-25T00:43:29.127 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:43:29.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117028 -lt 115964117028 2024-03-25T00:43:29.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:29.526 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757274 2024-03-25T00:43:29.526 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:29.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:43:29.528 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757274 2024-03-25T00:43:29.528 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:29.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757274 2024-03-25T00:43:29.530 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 163208757274 2024-03-25T00:43:29.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757274' 2024-03-25T00:43:29.530 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:43:29.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757275 -lt 163208757274 2024-03-25T00:43:29.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:29.953 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:29.953 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-201863462932 2024-03-25T00:43:29.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:43:29.955 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-201863462932 2024-03-25T00:43:29.955 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:29.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462932 2024-03-25T00:43:29.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 201863462932' 2024-03-25T00:43:29.956 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 201863462932 2024-03-25T00:43:29.957 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:43:30.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462932 -lt 201863462932 2024-03-25T00:43:30.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:43:30.348 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-244813135880 2024-03-25T00:43:30.348 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:43:30.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:43:30.350 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-244813135880 2024-03-25T00:43:30.350 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:43:30.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135880 2024-03-25T00:43:30.351 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 244813135880 2024-03-25T00:43:30.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 244813135880' 2024-03-25T00:43:30.352 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:43:30.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135881 -lt 244813135880 2024-03-25T00:43:30.753 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:43:30.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:43:30.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:43:31.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:43:31.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:43:31.280 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:43:31.280 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:43:31.280 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:43:31.281 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:43:31.281 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:43:31.281 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:43:31.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T00:43:31.730 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:43:31.730 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:43:31.730 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:43:32.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T00:43:32.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T00:43:32.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T00:43:32.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T00:43:32.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T00:43:32.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T00:43:32.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T00:43:32.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:43:32.391 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:43:32.391 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:43:32.391 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:43:32.391 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:43:32.391 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:43:32.392 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:43:32.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T00:43:32.781 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:43:32.781 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:43:32.781 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:43:33.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T00:43:33.354 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T00:43:33.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T00:43:33.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T00:43:33.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T00:43:33.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T00:43:33.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T00:43:33.355 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T00:43:33.355 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T00:43:33.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1803 2024-03-25T00:43:33.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1803 '!=' null 2024-03-25T00:43:33.891 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-25T00:43:33.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T00:43:33.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T00:43:33.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T00:43:33.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:43:33.993 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:43:33.993 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:43:33.993 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:43:33.994 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:43:33.994 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:43:33.994 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:43:34.382 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:43:34.382 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:43:34.382 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:43:34.383 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:43:34.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:43:34.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:43:34.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:43:34.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:502: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2024-03-25T00:43:34.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:43:34.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:43:34.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-03-25T00:43:34.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:43:34.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:43:34.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-03-25T00:43:34.977 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:43:34.978 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:43:34.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2024-03-25T00:43:34.979 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:43:34.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:43:34.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-03-25T00:43:34.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:43:34.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:43:34.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-03-25T00:43:35.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:43:35.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:43:35.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2024-03-25T00:43:35.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:43:35.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:43:35.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-03-25T00:43:35.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:43:35.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:43:35.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-03-25T00:43:35.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:43:35.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:43:35.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2024-03-25T00:43:35.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:43:35.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-03-25T00:43:35.596 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' does not exist 2024-03-25T00:43:35.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-03-25T00:43:36.135 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile myprofile does not exist 2024-03-25T00:43:36.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-03-25T00:43:36.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:43:36.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:43:36.151 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:43:36.151 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:43:36.152 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:43:36.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:43:36.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:43:36.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:43:36.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:43:36.466 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:43:36.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:43:36.468 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:43:36.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:43:36.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:43:36.470 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:43:36.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:43:36.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:43:36.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:43:36.472 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:43:36.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:43:36.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:43:36.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:43:36.515 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:43:36.515 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:36.515 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:36.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:43:36.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:43:36.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:43:36.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-03-25T00:43:36.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-03-25T00:43:36.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-03-25T00:43:36.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-03-25T00:43:36.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:43:36.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:43:36.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:43:36.518 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:43:36.518 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:43:36.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:43:36.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:43:36.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:43:36.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:43:36.520 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:43:36.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:43:36.521 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:43:36.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:43:36.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:43:36.523 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:43:36.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:43:36.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:43:36.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:43:36.525 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:43:36.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:43:36.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:43:36.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:43:36.528 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:43:36.528 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:36.528 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:36.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:43:36.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:43:36.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:43:36.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-03-25T00:43:36.531 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:43:36.531 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:36.531 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:36.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81310 2024-03-25T00:43:36.534 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:43:36.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:43:36.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:43:36.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-03-25T00:43:36.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-03-25T00:43:36.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-03-25T00:43:36.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:43:36.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:43:36.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:43:36.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-03-25T00:43:36.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-03-25T00:43:37.028 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:43:37.028 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:43:37.028 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:43:37.029 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:43:37.029 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:37.029 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:37.029 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:43:37.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T00:43:37.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:43:37.248 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:43:37.248 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:43:37.248 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:43:37.248 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:43:37.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:43:37.249 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:43:37.249 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:43:37.249 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:43:37.250 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:43:37.250 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:37.250 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:37.250 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:43:37.250 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:43:37.250 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get fsid 2024-03-25T00:43:37.365 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:43:37.365 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:43:37.366 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:43:37.366 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:43:37.367 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:43:37.367 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:43:37.367 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:43:37.367 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:43:37.367 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:43:37.367 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:37.367 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:37.367 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:43:37.368 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:43:37.368 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get mon_host 2024-03-25T00:43:37.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-03-25T00:43:37.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-03-25T00:43:37.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:43:37.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:43:37.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:43:37.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-03-25T00:43:37.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:43:38.375 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:43:38.376 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:43:38.376 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:43:38.376 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:43:38.376 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:38.376 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:38.376 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:43:38.376 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:43:38.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:43:38.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-03-25T00:43:38.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T00:43:38.701 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' already exists 2024-03-25T00:43:38.717 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:43:39.720 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-03-25T00:43:39.721 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:43:39.721 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:43:39.722 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:43:39.722 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:39.722 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:39.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:43:39.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-03-25T00:43:39.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-mon.a.asok log flush 2024-03-25T00:43:39.823 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:43:39.835 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-03-25T00:43:39.837 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:43:37.069+0000 7f057b51ac80 0 load: jerasure load: lrc 2024-03-25T00:43:39.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2024-03-25T00:43:39.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2024-03-25T00:43:39.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:610: TEST_ec_recovery_unfound: local objname=myobject 2024-03-25T00:43:39.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_recovery_unfound: local lastobj=100 2024-03-25T00:43:39.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_recovery_unfound: local testobj=obj75 2024-03-25T00:43:39.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:43:39.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:616: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-03-25T00:43:39.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:43:39.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_recovery_unfound: setup_osds 5 2024-03-25T00:43:39.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-03-25T00:43:39.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-03-25T00:43:39.839 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-03-25T00:43:39.840 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-03-25T00:43:39.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:43:39.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-03-25T00:43:39.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:43:39.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:43:39.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:43:39.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:43:39.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-03-25T00:43:39.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:43:39.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:43:39.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:43:39.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:43:39.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-03-25T00:43:39.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-03-25T00:43:39.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:43:39.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:43:39.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:43:39.845 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:43:39.845 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:43:39.845 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:43:39.845 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:43:39.845 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:43:39.846 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:43:39.846 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:43:39.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:43:39.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:43:39.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:43:39.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:43:39.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:43:39.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:43:39.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:43:39.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:43:39.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:43:39.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:43:39.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:43:39.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:43:39.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-03-25T00:43:39.847 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:43:39.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bcf15574-2209-40de-a6a9-bcd09cb56e2f 2024-03-25T00:43:39.848 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 bcf15574-2209-40de-a6a9-bcd09cb56e2f 2024-03-25T00:43:39.848 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bcf15574-2209-40de-a6a9-bcd09cb56e2f' 2024-03-25T00:43:39.848 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:43:39.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7yABmelZQNBAAd2Sx4l53zf4E3buwzvaLJQ== 2024-03-25T00:43:39.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7yABmelZQNBAAd2Sx4l53zf4E3buwzvaLJQ=="}' 2024-03-25T00:43:39.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bcf15574-2209-40de-a6a9-bcd09cb56e2f -i td/test-erasure-eio/0/new.json 2024-03-25T00:43:40.220 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:43:40.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-03-25T00:43:40.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7yABmelZQNBAAd2Sx4l53zf4E3buwzvaLJQ== --osd-uuid bcf15574-2209-40de-a6a9-bcd09cb56e2f 2024-03-25T00:43:40.271 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:40.269+0000 7f1d7176e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:43:40.296 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:40.293+0000 7f1d7176e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:43:40.297 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:40.293+0000 7f1d7176e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:43:40.297 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:40.293+0000 7f1d7176e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:43:40.298 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:40.293+0000 7f1d7176e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:43:40.298 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:40.293+0000 7f1d7176e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:43:40.336 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:40.333+0000 7f1d7176e7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-03-25T00:43:46.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-03-25T00:43:46.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:43:46.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:43:46.311 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:43:46.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:43:46.865 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:43:46.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:43:46.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:43:46.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:43:46.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:43:46.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:43:46.907 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:46.901+0000 7f979c38e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:43:46.954 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:46.949+0000 7f979c38e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:43:46.957 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:46.953+0000 7f979c38e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:43:47.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:43:47.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:43:47.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:43:47.335 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:43:47.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:43:47.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:43:47.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:47.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:43:47.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:47.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:43:47.516 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:47.513+0000 7f979c38e7c0 -1 Falling back to public interface 2024-03-25T00:43:47.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:48.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:48.771 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:48.771 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:43:48.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:43:48.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:48.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:43:49.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:50.117 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:43:50.116+0000 7f979c38e7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:43:50.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:50.187 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:50.188 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:43:50.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:43:50.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:50.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:43:50.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:51.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:51.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:51.790 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:43:51.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:43:51.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:51.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:43:52.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:53.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:53.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:53.225 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:43:53.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:43:53.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:53.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:43:53.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:54.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:54.672 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:54.672 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:43:54.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:43:54.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:54.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:43:55.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:56.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:56.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:56.245 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:43:56.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:43:56.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:56.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:43:56.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:57.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:57.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:57.707 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:43:57.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:43:57.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:57.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:43:58.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:43:59.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:43:59.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:43:59.293 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:43:59.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:43:59.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:43:59.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:43:59.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:00.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:00.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:00.727 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:44:00.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:44:00.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:00.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:44:01.278 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:02.278 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:02.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:02.279 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:44:02.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:44:02.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:02.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:44:02.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:03.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:03.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:03.697 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:44:03.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:44:03.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:44:03.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:04.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:05.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:05.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:05.429 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:44:05.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:44:05.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:05.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:44:06.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:07.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:07.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:07.298 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:44:07.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:44:07.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:07.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:44:08.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:09.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:09.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:09.118 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:44:09.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:44:09.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:09.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:44:09.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:10.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:10.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:10.583 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:44:10.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:44:10.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:10.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:44:11.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:12.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:12.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:12.412 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:44:12.412 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:44:12.412 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:12.412 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:44:13.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:14.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:14.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:14.170 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:44:14.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:44:14.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:14.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:44:14.725 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2805307598,v1:127.0.0.1:6803/2805307598] [v2:127.0.0.1:6804/2805307598,v1:127.0.0.1:6805/2805307598] exists,up bcf15574-2209-40de-a6a9-bcd09cb56e2f 2024-03-25T00:44:14.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:44:14.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:44:14.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:44:14.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:44:14.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:44:14.730 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:44:14.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:44:14.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:44:14.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:44:14.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:44:14.732 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:44:14.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47b341d1-00b9-4250-9e1a-36f18b7dc54c 2024-03-25T00:44:14.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 47b341d1-00b9-4250-9e1a-36f18b7dc54c' 2024-03-25T00:44:14.733 INFO:tasks.workunit.client.0.smithi080.stdout:add osd1 47b341d1-00b9-4250-9e1a-36f18b7dc54c 2024-03-25T00:44:14.733 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:44:14.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDeyABm+6OALRAAalgjflkOANFqH++Hgd+fNg== 2024-03-25T00:44:14.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDeyABm+6OALRAAalgjflkOANFqH++Hgd+fNg=="}' 2024-03-25T00:44:14.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47b341d1-00b9-4250-9e1a-36f18b7dc54c -i td/test-erasure-eio/1/new.json 2024-03-25T00:44:15.250 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:44:15.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-03-25T00:44:15.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDeyABm+6OALRAAalgjflkOANFqH++Hgd+fNg== --osd-uuid 47b341d1-00b9-4250-9e1a-36f18b7dc54c 2024-03-25T00:44:15.311 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:15.312+0000 7fb58cfb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:15.313 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:15.312+0000 7fb58cfb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:15.314 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:15.312+0000 7fb58cfb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:15.315 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:15.312+0000 7fb58cfb67c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:44:15.315 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:15.312+0000 7fb58cfb67c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:44:15.315 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:15.312+0000 7fb58cfb67c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:44:15.351 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:15.348+0000 7fb58cfb67c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-03-25T00:44:21.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-03-25T00:44:21.749 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:44:21.750 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd1 key to auth repository 2024-03-25T00:44:21.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T00:44:21.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:44:22.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T00:44:22.313 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.1 2024-03-25T00:44:22.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:44:22.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:44:22.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:44:22.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:44:22.353 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:22.352+0000 7f243b5187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:22.376 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:22.376+0000 7f243b5187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:22.378 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:22.376+0000 7f243b5187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:22.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T00:44:22.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:44:22.798 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:44:22.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:44:22.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:44:22.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:44:22.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:22.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:44:22.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:22.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:22.929 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:22.928+0000 7f243b5187c0 -1 Falling back to public interface 2024-03-25T00:44:23.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:24.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:24.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:24.249 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:44:24.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:44:24.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:24.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:24.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:25.479 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:25.476+0000 7f243b5187c0 -1 osd.1 0 log_to_monitors true 2024-03-25T00:44:25.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:25.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:25.681 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:44:25.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:44:25.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:25.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:26.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:27.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:27.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:27.267 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:44:27.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:44:27.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:27.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:28.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:28.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:28.831 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:44:28.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:44:28.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:28.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:29.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:30.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:30.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:30.388 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:44:30.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:44:30.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:30.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:30.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:31.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:31.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:31.861 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:44:31.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:44:31.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:31.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:32.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:33.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:33.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:33.524 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:44:33.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:44:33.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:33.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:34.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:35.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:35.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:35.132 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:44:35.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:44:35.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:35.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:35.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:36.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:36.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:36.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:44:36.784 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:44:36.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:36.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:37.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:38.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:38.401 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:38.401 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:44:38.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:44:38.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:38.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:39.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:40.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:40.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:40.120 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:44:40.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:44:40.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:40.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:41.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:42.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:42.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:42.097 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:44:42.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:44:42.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:42.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:43.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:44.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:44.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:44.045 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:44:44.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:44:44.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:44.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:44.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:45.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:45.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:45.887 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:44:45.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:44:45.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:45.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:46.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:47.418 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:47.418 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:47.419 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:44:47.419 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:44:47.419 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:47.419 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:48.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:49.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:49.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:49.600 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:44:49.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:44:49.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:49.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:44:50.193 INFO:tasks.workunit.client.0.smithi080.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2938572923,v1:127.0.0.1:6811/2938572923] [v2:127.0.0.1:6812/2938572923,v1:127.0.0.1:6813/2938572923] exists,up 47b341d1-00b9-4250-9e1a-36f18b7dc54c 2024-03-25T00:44:50.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:44:50.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:44:50.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:44:50.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:44:50.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-03-25T00:44:50.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:44:50.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:44:50.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:44:50.196 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:44:50.197 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:44:50.197 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:44:50.197 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:44:50.197 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:44:50.197 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:44:50.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:44:50.198 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=29c9759e-8c72-459f-9b75-7033a9f0b77b 2024-03-25T00:44:50.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 29c9759e-8c72-459f-9b75-7033a9f0b77b' 2024-03-25T00:44:50.200 INFO:tasks.workunit.client.0.smithi080.stdout:add osd2 29c9759e-8c72-459f-9b75-7033a9f0b77b 2024-03-25T00:44:50.200 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:44:50.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACyQBmXlyLDRAA/CnRLM3VxWhr07r8KHykNA== 2024-03-25T00:44:50.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACyQBmXlyLDRAA/CnRLM3VxWhr07r8KHykNA=="}' 2024-03-25T00:44:50.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 29c9759e-8c72-459f-9b75-7033a9f0b77b -i td/test-erasure-eio/2/new.json 2024-03-25T00:44:50.702 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:44:50.717 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-03-25T00:44:50.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACyQBmXlyLDRAA/CnRLM3VxWhr07r8KHykNA== --osd-uuid 29c9759e-8c72-459f-9b75-7033a9f0b77b 2024-03-25T00:44:50.761 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:50.759+0000 7f9de977f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:50.763 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:50.759+0000 7f9de977f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:50.764 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:50.763+0000 7f9de977f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:50.764 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:50.763+0000 7f9de977f7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:44:50.764 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:50.763+0000 7f9de977f7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:44:50.765 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:50.763+0000 7f9de977f7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:44:50.803 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:50.799+0000 7f9de977f7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-03-25T00:44:57.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-03-25T00:44:57.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:44:57.067 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd2 key to auth repository 2024-03-25T00:44:57.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T00:44:57.067 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:44:57.629 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:44:57.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T00:44:57.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:44:57.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:44:57.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:44:57.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:44:57.672 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:57.671+0000 7f0fea9fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:57.674 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:57.671+0000 7f0fea9fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:57.675 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:57.675+0000 7f0fea9fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:44:58.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T00:44:58.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:44:58.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:44:58.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:44:58.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:44:58.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:58.034 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:44:58.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:44:58.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:58.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:44:58.213 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:44:58.211+0000 7f0fea9fb7c0 -1 Falling back to public interface 2024-03-25T00:44:58.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:44:59.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:44:59.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:44:59.496 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:44:59.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:44:59.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:44:59.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:44:59.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:00.772 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:00.771+0000 7f0fea9fb7c0 -1 osd.2 0 log_to_monitors true 2024-03-25T00:45:00.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:00.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:00.926 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:45:00.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:45:00.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:00.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:01.463 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:02.465 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:02.465 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:02.465 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:45:02.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:45:02.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:02.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:02.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:03.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:03.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:03.927 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:45:03.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:45:03.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:03.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:04.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:05.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:05.355 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:05.355 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:45:05.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:45:05.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:05.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:05.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:06.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:06.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:06.879 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:45:06.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:45:06.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:06.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:07.371 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:08.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:08.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:08.373 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:45:08.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:45:08.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:08.374 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:08.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:09.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:09.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:09.907 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:45:09.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:45:09.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:09.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:10.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:11.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:11.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:11.341 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:45:11.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:45:11.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:11.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:11.896 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:12.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:12.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:12.899 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:45:12.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:45:12.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:12.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:13.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:14.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:14.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:14.600 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:45:14.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:45:14.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:14.600 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:15.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:16.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:16.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:16.722 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:45:16.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:45:16.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:16.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:17.465 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:18.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:18.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:18.467 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:45:18.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:45:18.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:18.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:19.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:20.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:20.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:20.324 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:45:20.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:45:20.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:20.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:21.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:22.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:22.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:22.324 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:45:22.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:45:22.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:22.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:23.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:24.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:24.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:24.136 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:45:24.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:45:24.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:24.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:45:24.678 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1904860206,v1:127.0.0.1:6819/1904860206] [v2:127.0.0.1:6820/1904860206,v1:127.0.0.1:6821/1904860206] exists,up 29c9759e-8c72-459f-9b75-7033a9f0b77b 2024-03-25T00:45:24.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:45:24.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:45:24.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:45:24.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:45:24.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:45:24.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:45:24.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:45:24.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:45:24.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:45:24.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:45:24.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:45:24.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:45:24.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:45:24.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:45:24.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:45:24.685 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:45:24.685 INFO:tasks.workunit.client.0.smithi080.stdout:add osd3 8403eb0c-5c0a-40e3-a37f-78876d9a59b2 2024-03-25T00:45:24.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8403eb0c-5c0a-40e3-a37f-78876d9a59b2 2024-03-25T00:45:24.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 8403eb0c-5c0a-40e3-a37f-78876d9a59b2' 2024-03-25T00:45:24.685 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:45:24.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAkyQBmXhV+KhAAJS1EnpLpfqgU4LHb0vbbzw== 2024-03-25T00:45:24.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAkyQBmXhV+KhAAJS1EnpLpfqgU4LHb0vbbzw=="}' 2024-03-25T00:45:24.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8403eb0c-5c0a-40e3-a37f-78876d9a59b2 -i td/test-erasure-eio/3/new.json 2024-03-25T00:45:25.780 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:45:25.801 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-03-25T00:45:25.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkyQBmXhV+KhAAJS1EnpLpfqgU4LHb0vbbzw== --osd-uuid 8403eb0c-5c0a-40e3-a37f-78876d9a59b2 2024-03-25T00:45:25.846 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:25.842+0000 7fe9ce0187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:45:25.887 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:25.882+0000 7fe9ce0187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:45:25.888 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:25.886+0000 7fe9ce0187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:45:25.889 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:25.886+0000 7fe9ce0187c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:45:25.889 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:25.886+0000 7fe9ce0187c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:45:25.889 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:25.886+0000 7fe9ce0187c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:45:25.978 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:25.974+0000 7fe9ce0187c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-03-25T00:45:32.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-03-25T00:45:32.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:45:32.442 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd3 key to auth repository 2024-03-25T00:45:32.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T00:45:32.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:45:32.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T00:45:32.991 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.3 2024-03-25T00:45:32.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:45:32.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:45:32.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:45:32.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:45:33.031 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:33.026+0000 7f7751c167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:45:33.033 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:33.030+0000 7f7751c167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:45:33.034 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:33.030+0000 7f7751c167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:45:33.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T00:45:33.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:45:33.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:45:33.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:45:33.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:45:33.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:33.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:45:33.415 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:45:33.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:33.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:33.601 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:33.598+0000 7f7751c167c0 -1 Falling back to public interface 2024-03-25T00:45:33.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:34.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:34.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:34.949 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:45:34.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:45:34.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:34.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:35.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:36.119 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:45:36.114+0000 7f7751c167c0 -1 osd.3 0 log_to_monitors true 2024-03-25T00:45:36.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:36.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:36.391 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:45:36.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:45:36.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:36.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:37.025 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:38.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:38.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:38.027 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:45:38.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:45:38.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:38.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:38.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:39.548 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:39.548 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:39.548 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:45:39.548 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:45:39.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:39.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:40.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:41.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:41.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:41.230 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:45:41.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:45:41.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:41.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:41.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:42.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:42.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:42.786 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:45:42.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:45:42.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:42.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:43.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:44.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:44.261 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:44.262 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:45:44.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:45:44.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:44.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:44.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:45.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:45.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:45.701 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:45:45.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:45:45.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:45.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:46.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:47.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:47.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:47.269 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:45:47.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:45:47.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:47.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:47.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:48.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:48.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:48.817 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:45:48.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:45:48.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:48.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:49.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:50.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:50.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:50.298 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:45:50.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:45:50.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:50.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:50.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:51.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:51.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:51.853 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:45:51.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:45:51.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:51.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:52.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:53.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:53.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:53.715 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:45:53.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:45:53.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:53.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:54.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:55.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:55.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:55.555 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:45:55.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:45:55.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:55.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:56.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:57.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:57.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:57.409 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:45:57.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:45:57.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:57.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:45:58.420 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:45:59.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:45:59.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:45:59.422 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:45:59.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:45:59.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:45:59.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:46:00.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:01.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:01.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:01.088 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:46:01.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:46:01.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:01.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:46:01.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:02.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:02.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:02.549 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:46:02.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:46:02.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:02.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:46:03.078 INFO:tasks.workunit.client.0.smithi080.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/552052779,v1:127.0.0.1:6827/552052779] [v2:127.0.0.1:6828/552052779,v1:127.0.0.1:6829/552052779] exists,up 8403eb0c-5c0a-40e3-a37f-78876d9a59b2 2024-03-25T00:46:03.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:46:03.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:46:03.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-03-25T00:46:03.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-03-25T00:46:03.082 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:46:03.082 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:46:03.082 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:46:03.082 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:46:03.082 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:46:03.082 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:46:03.082 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:46:03.082 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:46:03.082 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:46:03.082 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:46:03.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-03-25T00:46:03.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:46:03.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1afc2666-ec23-495d-8562-59abce7fcce1 2024-03-25T00:46:03.085 INFO:tasks.workunit.client.0.smithi080.stdout:add osd4 1afc2666-ec23-495d-8562-59abce7fcce1 2024-03-25T00:46:03.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 1afc2666-ec23-495d-8562-59abce7fcce1' 2024-03-25T00:46:03.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:46:03.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLyQBmrmDEBhAArX/9YWJXwxKaLiJjMeFNKQ== 2024-03-25T00:46:03.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLyQBmrmDEBhAArX/9YWJXwxKaLiJjMeFNKQ=="}' 2024-03-25T00:46:03.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1afc2666-ec23-495d-8562-59abce7fcce1 -i td/test-erasure-eio/4/new.json 2024-03-25T00:46:03.807 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:46:03.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-03-25T00:46:03.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLyQBmrmDEBhAArX/9YWJXwxKaLiJjMeFNKQ== --osd-uuid 1afc2666-ec23-495d-8562-59abce7fcce1 2024-03-25T00:46:03.868 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:03.865+0000 7f6e180ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:46:03.971 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:03.965+0000 7f6e180ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:46:03.972 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:03.969+0000 7f6e180ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:46:03.973 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:03.969+0000 7f6e180ff7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:46:03.973 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:03.969+0000 7f6e180ff7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:46:03.973 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:03.969+0000 7f6e180ff7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:46:04.073 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:04.069+0000 7f6e180ff7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-03-25T00:46:10.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-03-25T00:46:10.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:46:10.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-25T00:46:10.743 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd4 key to auth repository 2024-03-25T00:46:10.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:46:11.315 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.4 2024-03-25T00:46:11.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-25T00:46:11.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:46:11.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:46:11.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:46:11.316 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:46:11.358 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:11.349+0000 7f05ef9657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:46:11.402 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:11.397+0000 7f05ef9657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:46:11.404 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:11.401+0000 7f05ef9657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:46:11.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-25T00:46:11.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:46:11.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-25T00:46:11.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:46:11.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:46:11.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:11.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:46:11.742 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:46:11.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:11.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:11.937 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:11.933+0000 7f05ef9657c0 -1 Falling back to public interface 2024-03-25T00:46:12.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:13.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:13.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:13.223 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:46:13.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:46:13.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:13.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:13.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:14.559 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:46:14.553+0000 7f05ef9657c0 -1 osd.4 0 log_to_monitors true 2024-03-25T00:46:14.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:14.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:14.666 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:46:14.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:46:14.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:14.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:15.418 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:16.420 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:16.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:16.421 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:46:16.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:46:16.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:16.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:16.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:17.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:17.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:17.929 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:46:17.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:46:17.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:17.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:18.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:19.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:19.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:19.563 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:46:19.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:46:19.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:19.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:20.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:21.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:21.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:21.095 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:46:21.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:46:21.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:21.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:21.541 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:22.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:22.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:22.543 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:46:22.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:46:22.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:22.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:23.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:24.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:24.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:24.014 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:46:24.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:46:24.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:24.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:24.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:25.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:25.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:25.517 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:46:25.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:46:25.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:25.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:25.968 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:26.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:26.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:26.971 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:46:26.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:46:26.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:26.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:27.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:28.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:28.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:28.537 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:46:28.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:46:28.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:28.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:28.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:29.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:29.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:29.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:46:29.972 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:46:29.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:29.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:30.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:31.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:31.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:31.833 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:46:31.833 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:46:31.833 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:31.833 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:32.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:33.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:33.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:33.679 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:46:33.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:46:33.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:33.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:34.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:35.747 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:35.747 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:35.747 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:46:35.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:46:35.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:35.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:36.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:37.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:37.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:37.722 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:46:37.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:46:37.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:37.723 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:38.420 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:39.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:39.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:39.422 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:46:39.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:46:39.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:39.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:39.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:46:40.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:46:40.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:46:40.894 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:46:40.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:46:40.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:46:40.895 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:46:41.473 INFO:tasks.workunit.client.0.smithi080.stdout:osd.4 up in weight 1 up_from 38 up_thru 40 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1013922853,v1:127.0.0.1:6835/1013922853] [v2:127.0.0.1:6836/1013922853,v1:127.0.0.1:6837/1013922853] exists,up 1afc2666-ec23-495d-8562-59abce7fcce1 2024-03-25T00:46:41.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:46:41.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:46:41.475 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:46:41.475 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-03-25T00:46:41.475 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:46:41.475 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:46:41.475 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:46:41.475 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:46:41.476 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:46:41.476 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.0.asok 2024-03-25T00:46:41.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-03-25T00:46:41.476 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.0.asok log flush 2024-03-25T00:46:41.577 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:46:41.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-03-25T00:46:41.594 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:43:47.517+0000 7f979c38e7c0 0 load: jerasure load: lrc 2024-03-25T00:46:41.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:46:41.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:621: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2024-03-25T00:46:41.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-03-25T00:46:41.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:46:41.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-03-25T00:46:41.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-03-25T00:46:41.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-03-25T00:46:41.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-03-25T00:46:41.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-03-25T00:46:41.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-03-25T00:46:42.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-03-25T00:46:42.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-03-25T00:46:43.225 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' already exists 2024-03-25T00:46:43.240 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:46:44.242 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-03-25T00:46:44.242 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:46:44.242 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:46:44.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:46:44.243 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:46:44.244 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:46:44.244 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:46:44.245 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:46:44.245 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:46:44.245 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:46:44.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:46:44.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:46:44.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:46:44.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:46:44.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:46:44.370 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:46:44.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:46:44.889 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:46:44.889 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:46:44.889 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:46:44.889 INFO:tasks.workunit.client.0.smithi080.stderr:4' 2024-03-25T00:46:44.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:46:44.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:46:44.890 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:46:45.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803810 2024-03-25T00:46:45.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803810 2024-03-25T00:46:45.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810' 2024-03-25T00:46:45.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:46:45.202 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:46:45.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574879 2024-03-25T00:46:45.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574879 2024-03-25T00:46:45.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810 1-55834574879' 2024-03-25T00:46:45.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:46:45.361 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:46:45.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378646 2024-03-25T00:46:45.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378646 2024-03-25T00:46:45.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810 1-55834574879 2-81604378646' 2024-03-25T00:46:45.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:46:45.526 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:46:45.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117007 2024-03-25T00:46:45.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117007 2024-03-25T00:46:45.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810 1-55834574879 2-81604378646 3-115964117007' 2024-03-25T00:46:45.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:46:45.700 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:46:45.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757256 2024-03-25T00:46:45.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757256 2024-03-25T00:46:45.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810 1-55834574879 2-81604378646 3-115964117007 4-163208757256' 2024-03-25T00:46:45.856 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:46:45.857 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803810 2024-03-25T00:46:45.857 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:46:45.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:46:45.859 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803810 2024-03-25T00:46:45.860 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:46:45.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803810 2024-03-25T00:46:45.861 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803810 2024-03-25T00:46:45.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803810' 2024-03-25T00:46:45.861 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:46:46.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803808 -lt 25769803810 2024-03-25T00:46:46.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:46:47.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:46:47.275 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:46:47.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803810 -lt 25769803810 2024-03-25T00:46:47.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:46:47.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574879 2024-03-25T00:46:47.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:46:47.724 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:46:47.725 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574879 2024-03-25T00:46:47.725 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:46:47.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574879 2024-03-25T00:46:47.726 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 55834574879 2024-03-25T00:46:47.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574879' 2024-03-25T00:46:47.727 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:46:48.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574879 -lt 55834574879 2024-03-25T00:46:48.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:46:48.206 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378646 2024-03-25T00:46:48.206 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:46:48.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:46:48.208 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378646 2024-03-25T00:46:48.208 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:46:48.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378646 2024-03-25T00:46:48.209 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 81604378646 2024-03-25T00:46:48.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378646' 2024-03-25T00:46:48.209 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:46:48.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378646 -lt 81604378646 2024-03-25T00:46:48.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:46:48.610 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117007 2024-03-25T00:46:48.610 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:46:48.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:46:48.613 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117007 2024-03-25T00:46:48.613 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:46:48.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117007 2024-03-25T00:46:48.614 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 115964117007 2024-03-25T00:46:48.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117007' 2024-03-25T00:46:48.615 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:46:49.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117007 -lt 115964117007 2024-03-25T00:46:49.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:46:49.038 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:46:49.038 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757256 2024-03-25T00:46:49.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:46:49.039 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757256 2024-03-25T00:46:49.039 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:46:49.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757256 2024-03-25T00:46:49.040 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 163208757256 2024-03-25T00:46:49.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757256' 2024-03-25T00:46:49.041 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:46:49.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757256 -lt 163208757256 2024-03-25T00:46:49.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:46:49.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:46:49.459 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:46:50.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:46:50.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:46:50.016 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:46:50.016 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:46:50.016 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:46:50.016 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:46:50.016 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:46:50.016 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:46:50.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:46:50.423 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:46:50.424 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:46:50.424 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:46:50.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:46:50.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:46:50.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:46:50.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-03-25T00:46:51.407 INFO:tasks.workunit.client.0.smithi080.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T00:46:51.408 INFO:tasks.workunit.client.0.smithi080.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:46:44.884719+0000 0'0 46:15 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-03-25T00:46:42.990182+0000 0'0 2024-03-25T00:46:42.990182+0000 0 0 periodic scrub scheduled @ 2024-03-26T02:09:41.300317+0000 0 0 2024-03-25T00:46:51.408 INFO:tasks.workunit.client.0.smithi080.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:46:41.852187+0000 0'0 46:19 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T04:53:09.090843+0000 0 0 2024-03-25T00:46:51.408 INFO:tasks.workunit.client.0.smithi080.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:46:42.020853+0000 0'0 46:104 [0,4,1] 0 [0,4,1] 0 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T11:16:31.464358+0000 0 0 2024-03-25T00:46:51.408 INFO:tasks.workunit.client.0.smithi080.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:46:42.020689+0000 0'0 46:19 [4,3,0] 4 [4,3,0] 4 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T06:40:52.842027+0000 0 0 2024-03-25T00:46:51.408 INFO:tasks.workunit.client.0.smithi080.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:45:25.443733+0000 0'0 46:73 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T02:54:32.252798+0000 0 0 2024-03-25T00:46:51.408 INFO:tasks.workunit.client.0.smithi080.stdout: 2024-03-25T00:46:51.408 INFO:tasks.workunit.client.0.smithi080.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T00:46:51.408 INFO:tasks.workunit.client.0.smithi080.stderr:dumped pgs 2024-03-25T00:46:51.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:626: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-03-25T00:46:51.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:46:51.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:46:51.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-03-25T00:46:51.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:46:51.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:46:51.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:46:51.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:46:51.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:46:51.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:46:51.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:46:51.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:46:51.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:51.625 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2024-03-25T00:46:51.625 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:46:51.625 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-03-25T00:46:51.626 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:46:51.626 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:46:52.036 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:46:52.036 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:46:52.036 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:46:52.036 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:46:52.036 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-25T00:46:52.036 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-03-25T00:46:52.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-03-25T00:46:52.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: local -a initial_osds 2024-03-25T00:46:52.036 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local last_osd=2 2024-03-25T00:46:52.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-03-25T00:46:52.037 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:46:52.037 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:46:52.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:46:52.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:46:52.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:46:52.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:46:52.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: ceph osd down 2 2024-03-25T00:46:53.238 INFO:tasks.workunit.client.0.smithi080.stderr:osd.2 is already down. 2024-03-25T00:46:53.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:632: TEST_ec_recovery_unfound: ceph osd out 2 2024-03-25T00:46:53.883 INFO:tasks.workunit.client.0.smithi080.stderr:osd.2 is already out. 2024-03-25T00:46:53.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:634: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-03-25T00:46:54.420 INFO:tasks.workunit.client.0.smithi080.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T00:46:54.421 INFO:tasks.workunit.client.0.smithi080.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 peering 2024-03-25T00:46:52.348463+0000 46'1 46:17 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-03-25T00:46:42.990182+0000 0'0 2024-03-25T00:46:42.990182+0000 0 0 no scrub is scheduled 0 0 2024-03-25T00:46:54.421 INFO:tasks.workunit.client.0.smithi080.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:46:41.852187+0000 0'0 46:19 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T04:53:09.090843+0000 0 0 2024-03-25T00:46:54.421 INFO:tasks.workunit.client.0.smithi080.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:46:42.020853+0000 0'0 49:110 [0,4,1] 0 [0,4,1] 0 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T11:16:31.464358+0000 0 0 2024-03-25T00:46:54.421 INFO:tasks.workunit.client.0.smithi080.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:46:42.020689+0000 0'0 46:19 [4,3,0] 4 [4,3,0] 4 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T06:40:52.842027+0000 0 0 2024-03-25T00:46:54.421 INFO:tasks.workunit.client.0.smithi080.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:45:25.443733+0000 0'0 46:73 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T02:54:32.252798+0000 0 0 2024-03-25T00:46:54.421 INFO:tasks.workunit.client.0.smithi080.stdout: 2024-03-25T00:46:54.421 INFO:tasks.workunit.client.0.smithi080.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T00:46:54.421 INFO:tasks.workunit.client.0.smithi080.stderr:dumped pgs 2024-03-25T00:46:54.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-03-25T00:46:54.439 INFO:tasks.workunit.client.0.smithi080.stderr:4+0 records in 2024-03-25T00:46:54.439 INFO:tasks.workunit.client.0.smithi080.stderr:4+0 records out 2024-03-25T00:46:54.439 INFO:tasks.workunit.client.0.smithi080.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000275634 s, 14.9 MB/s 2024-03-25T00:46:54.440 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: seq 1 100 2024-03-25T00:46:54.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:54.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:54.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:54.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:55.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:55.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:55.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:55.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:55.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:55.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:55.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:55.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:56.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:56.129 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:56.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:56.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:56.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:56.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:56.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:56.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:57.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:57.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:57.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:57.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:57.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:57.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:58.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:58.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:58.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:58.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:58.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:58.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:58.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:58.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:58.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:58.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:59.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:59.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:59.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:59.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-03-25T00:46:59.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:46:59.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:00.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:00.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:00.242 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:00.242 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:00.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:00.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:00.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:00.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:00.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:00.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:01.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:01.275 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:01.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:01.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:01.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:01.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:02.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:02.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:02.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:02.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:02.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:02.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:02.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:02.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:02.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:02.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:03.200 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:03.200 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:03.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:03.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:03.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:03.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:03.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:03.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:04.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:04.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:04.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:04.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:04.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:04.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:04.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:04.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:05.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:05.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:05.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:05.484 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:05.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:05.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:05.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:05.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:06.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:06.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:06.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:06.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:07.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:07.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:07.320 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:07.320 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:07.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:07.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:07.807 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:07.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:08.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:08.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:08.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:08.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:08.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:08.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:08.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:08.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:09.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:09.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:09.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:09.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:09.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:09.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:09.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:09.807 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:10.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:10.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:10.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:10.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:10.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:10.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:10.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:10.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:11.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:11.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:11.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:11.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:11.603 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:11.603 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:11.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:11.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:12.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:12.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:12.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:12.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:12.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:12.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:12.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:12.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:13.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:13.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:13.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:13.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:13.618 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:13.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:13.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:13.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:14.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:14.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:14.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:14.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:14.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:14.643 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:14.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:14.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:15.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:15.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:15.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:15.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:15.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:15.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:15.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:15.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:16.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:16.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:16.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:16.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:16.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:16.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:16.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:16.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:17.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:17.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:17.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:17.448 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:17.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:17.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:17.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:17.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:18.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:18.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:18.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:18.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:18.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:18.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:18.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:18.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:19.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:19.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:19.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:19.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:19.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:19.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:20.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:20.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-03-25T00:47:20.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2024-03-25T00:47:20.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-03-25T00:47:20.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-25T00:47:20.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-25T00:47:20.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-25T00:47:20.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-03-25T00:47:20.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-25T00:47:20.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj75 2024-03-25T00:47:20.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-25T00:47:20.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-03-25T00:47:20.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-25T00:47:20.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-25T00:47:20.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-25T00:47:20.254 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj75 2024-03-25T00:47:20.254 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:47:20.254 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-03-25T00:47:20.255 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-03-25T00:47:20.255 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:47:20.680 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:47:20.681 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:47:20.681 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:47:20.681 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:47:20.681 INFO:tasks.workunit.client.0.smithi080.stderr:2147483647' 2024-03-25T00:47:20.681 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-03-25T00:47:20.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-03-25T00:47:20.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-25T00:47:20.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-03-25T00:47:20.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-03-25T00:47:20.682 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-03-25T00:47:20.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-25T00:47:20.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-03-25T00:47:20.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T00:47:20.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-03-25T00:47:20.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-25T00:47:20.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-25T00:47:20.683 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T00:47:20.684 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-03-25T00:47:20.684 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-03-25T00:47:20.684 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-03-25T00:47:20.684 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:47:20.684 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:47:20.684 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:47:20.684 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.3.asok 2024-03-25T00:47:20.685 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81310/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-03-25T00:47:20.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T00:47:20.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-25T00:47:20.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-25T00:47:20.803 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-03-25T00:47:20.804 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-03-25T00:47:20.804 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-03-25T00:47:20.804 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:47:20.805 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:47:20.805 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:47:20.805 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.3.asok 2024-03-25T00:47:20.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-25T00:47:20.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2024-03-25T00:47:20.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:643: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2024-03-25T00:47:20.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-03-25T00:47:20.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-25T00:47:20.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-25T00:47:20.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-25T00:47:20.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-03-25T00:47:20.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-25T00:47:20.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj75 2024-03-25T00:47:20.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-25T00:47:20.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-03-25T00:47:20.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-25T00:47:20.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-25T00:47:20.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-25T00:47:20.933 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj75 2024-03-25T00:47:20.933 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:47:20.933 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-03-25T00:47:20.933 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:47:20.933 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-03-25T00:47:21.337 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:47:21.337 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:47:21.337 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:47:21.337 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:47:21.337 INFO:tasks.workunit.client.0.smithi080.stderr:2147483647' 2024-03-25T00:47:21.337 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-03-25T00:47:21.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-03-25T00:47:21.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-25T00:47:21.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-25T00:47:21.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-03-25T00:47:21.338 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/1/type 2024-03-25T00:47:21.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-25T00:47:21.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-25T00:47:21.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T00:47:21.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-25T00:47:21.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-25T00:47:21.339 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-25T00:47:21.340 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T00:47:21.340 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-25T00:47:21.340 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-25T00:47:21.340 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-25T00:47:21.341 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:47:21.341 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:47:21.341 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:47:21.341 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.1.asok 2024-03-25T00:47:21.341 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81310/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-25T00:47:21.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T00:47:21.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-25T00:47:21.465 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-25T00:47:21.465 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-25T00:47:21.465 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-25T00:47:21.465 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-25T00:47:21.466 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:47:21.466 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:47:21.466 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:47:21.466 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.1.asok 2024-03-25T00:47:21.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-25T00:47:21.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2024-03-25T00:47:21.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:645: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2024-03-25T00:47:21.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:47:21.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:47:21.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-25T00:47:21.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:47:21.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:47:21.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:47:21.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:47:21.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:47:21.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:47:21.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:47:21.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:47:21.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:47:21.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:47:21.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:47:21.590 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:47:21.590 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:47:21.590 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:47:21.590 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:47:21.590 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:47:21.590 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:47:21.590 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:47:21.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:47:21.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:47:21.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:47:21.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:47:21.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:47:21.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:47:21.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:47:21.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:47:21.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:47:21.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:47:21.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:47:21.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-25T00:47:21.592 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:47:21.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:47:21.593 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-03-25T00:47:21.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-25T00:47:21.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:47:21.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:47:21.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:47:21.626 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:47:21.623+0000 7f24272db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:47:21.629 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:47:21.627+0000 7f24272db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:47:21.630 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:47:21.627+0000 7f24272db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:47:21.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-25T00:47:21.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:47:21.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:47:21.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:47:21.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:47:21.986 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:47:21.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:47:21.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:47:21.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:47:21.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:47:22.182 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:47:22.179+0000 7f24272db7c0 -1 Falling back to public interface 2024-03-25T00:47:22.413 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:47:23.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:47:23.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:47:23.415 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:47:23.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:47:23.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:47:23.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:47:23.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:47:24.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:47:24.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:47:24.841 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:47:24.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:47:24.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:47:24.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:47:25.198 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:47:25.195+0000 7f24272db7c0 -1 osd.2 46 log_to_monitors true 2024-03-25T00:47:25.273 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:47:26.278 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:47:26.278 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:47:26.278 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:47:26.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:47:26.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:47:26.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:47:26.712 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up out weight 0 up_from 54 up_thru 19 down_at 47 last_clean_interval [19,46) [v2:127.0.0.1:6818/576697540,v1:127.0.0.1:6819/576697540] [v2:127.0.0.1:6820/576697540,v1:127.0.0.1:6821/576697540] exists,up 29c9759e-8c72-459f-9b75-7033a9f0b77b 2024-03-25T00:47:26.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:47:26.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:47:26.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:47:26.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: ceph osd in 2 2024-03-25T00:47:27.312 INFO:tasks.workunit.client.0.smithi080.stderr:osd.2 is already in. 2024-03-25T00:47:27.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: sleep 15 2024-03-25T00:47:42.330 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: seq 1 100 2024-03-25T00:47:42.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2024-03-25T00:47:42.332 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: get_state 2.0 2024-03-25T00:47:42.332 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-03-25T00:47:42.332 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-03-25T00:47:42.333 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-03-25T00:47:42.333 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-03-25T00:47:42.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2024-03-25T00:47:42.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2024-03-25T00:47:42.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: grep recovery_unfound 2024-03-25T00:47:42.741 INFO:tasks.workunit.client.0.smithi080.stdout:active+recovery_unfound+degraded 2024-03-25T00:47:42.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2024-03-25T00:47:42.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: break 2024-03-25T00:47:42.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:660: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-03-25T00:47:43.136 INFO:tasks.workunit.client.0.smithi080.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-25T00:47:43.136 INFO:tasks.workunit.client.0.smithi080.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-03-25T00:47:31.771435+0000 52'101 57:349 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-03-25T00:46:42.990182+0000 0'0 2024-03-25T00:46:42.990182+0000 0 0 no scrub is scheduled 0 0 2024-03-25T00:47:43.136 INFO:tasks.workunit.client.0.smithi080.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:47:28.936756+0000 0'0 57:54 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T01:59:40.135449+0000 0 0 2024-03-25T00:47:43.136 INFO:tasks.workunit.client.0.smithi080.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:46:42.020853+0000 0'0 57:126 [0,4,1] 0 [0,4,1] 0 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T11:16:31.464358+0000 0 0 2024-03-25T00:47:43.136 INFO:tasks.workunit.client.0.smithi080.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:46:42.020689+0000 0'0 57:41 [4,3,0] 4 [4,3,0] 4 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T06:40:52.842027+0000 0 0 2024-03-25T00:47:43.137 INFO:tasks.workunit.client.0.smithi080.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-25T00:47:28.937000+0000 0'0 57:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-25T00:43:38.602393+0000 0'0 2024-03-25T00:43:38.602393+0000 0 0 periodic scrub scheduled @ 2024-03-26T09:14:02.152779+0000 0 0 2024-03-25T00:47:43.137 INFO:tasks.workunit.client.0.smithi080.stdout: 2024-03-25T00:47:43.137 INFO:tasks.workunit.client.0.smithi080.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-25T00:47:43.137 INFO:tasks.workunit.client.0.smithi080.stderr:dumped pgs 2024-03-25T00:47:43.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:661: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-03-25T00:47:43.295 INFO:tasks.workunit.client.0.smithi080.stdout:{ 2024-03-25T00:47:43.295 INFO:tasks.workunit.client.0.smithi080.stdout: "num_missing": 1, 2024-03-25T00:47:43.295 INFO:tasks.workunit.client.0.smithi080.stdout: "num_unfound": 1, 2024-03-25T00:47:43.295 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": [ 2024-03-25T00:47:43.295 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.295 INFO:tasks.workunit.client.0.smithi080.stdout: "oid": { 2024-03-25T00:47:43.295 INFO:tasks.workunit.client.0.smithi080.stdout: "oid": "obj75", 2024-03-25T00:47:43.295 INFO:tasks.workunit.client.0.smithi080.stdout: "key": "", 2024-03-25T00:47:43.295 INFO:tasks.workunit.client.0.smithi080.stdout: "snapid": -2, 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: "hash": 3565005477, 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: "max": 0, 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: "pool": 2, 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: "namespace": "" 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: "need": "52'76", 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: "have": "0'0", 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: "flags": "none", 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: "locations": [ 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: ] 2024-03-25T00:47:43.296 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.297 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.297 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "NotRecovering", 2024-03-25T00:47:43.297 INFO:tasks.workunit.client.0.smithi080.stdout: "available_might_have_unfound": true, 2024-03-25T00:47:43.297 INFO:tasks.workunit.client.0.smithi080.stdout: "might_have_unfound": [], 2024-03-25T00:47:43.297 INFO:tasks.workunit.client.0.smithi080.stdout: "more": false 2024-03-25T00:47:43.297 INFO:tasks.workunit.client.0.smithi080.stdout:} 2024-03-25T00:47:43.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: ceph pg 2.0 query 2024-03-25T00:47:43.447 INFO:tasks.workunit.client.0.smithi080.stdout:{ 2024-03-25T00:47:43.447 INFO:tasks.workunit.client.0.smithi080.stdout: "snap_trimq": "[]", 2024-03-25T00:47:43.447 INFO:tasks.workunit.client.0.smithi080.stdout: "snap_trimq_len": 0, 2024-03-25T00:47:43.447 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "active+recovery_unfound+degraded", 2024-03-25T00:47:43.447 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch": 57, 2024-03-25T00:47:43.447 INFO:tasks.workunit.client.0.smithi080.stdout: "up": [ 2024-03-25T00:47:43.447 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.447 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: "acting": [ 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.448 INFO:tasks.workunit.client.0.smithi080.stdout: "acting_recovery_backfill": [ 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "1(1)", 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "2(4)", 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "3(0)", 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "info": { 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "pgid": "2.0s0", 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "shared": "0", 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "last_update": "52'101", 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "last_complete": "52'75", 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "log_tail": "0'0", 2024-03-25T00:47:43.449 INFO:tasks.workunit.client.0.smithi080.stdout: "last_user_version": 101, 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "last_backfill": "MAX", 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [], 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "history": { 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_created": 44, 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_pool_created": 44, 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 56, 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_started": 55, 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 45, 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_clean": 44, 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_split": 0, 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_marked_full": 0, 2024-03-25T00:47:43.450 INFO:tasks.workunit.client.0.smithi080.stdout: "same_up_since": 55, 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "same_interval_since": 55, 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "same_primary_since": 44, 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "prior_readable_until_ub": 0 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "stats": { 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "version": "52'101", 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_seq": 350, 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_epoch": 57, 2024-03-25T00:47:43.451 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "active+recovery_unfound+degraded", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fresh": "2024-03-25T00:47:31.836553+0000", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "last_change": "2024-03-25T00:47:31.771435+0000", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "last_active": "2024-03-25T00:47:31.836553+0000", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "last_peered": "2024-03-25T00:47:31.836553+0000", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean": "2024-03-25T00:46:51.622310+0000", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_active": "2024-03-25T00:47:28.348960+0000", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_peered": "2024-03-25T00:47:28.348960+0000", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "last_unstale": "2024-03-25T00:47:31.836553+0000", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "last_undegraded": "2024-03-25T00:47:27.929186+0000", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fullsized": "2024-03-25T00:47:31.836553+0000", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "mapping_epoch": 55, 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "log_start": "0'0", 2024-03-25T00:47:43.452 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_start": "0'0", 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "created": 44, 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 45, 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "parent": "0.0", 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "parent_split_bits": 0, 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_scrubbed": 0, 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "log_size": 101, 2024-03-25T00:47:43.453 INFO:tasks.workunit.client.0.smithi080.stdout: "log_dups_size": 0, 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_size": 101, 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "stats_invalid": false, 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "dirty_stats_invalid": false, 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "omap_stats_invalid": false, 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_stats_invalid": false, 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_bytes_stats_invalid": false, 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "pin_stats_invalid": false, 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "manifest_stats_invalid": false, 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrimq_len": 0, 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_duration": 0, 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_schedule": "no scrub is scheduled", 2024-03-25T00:47:43.454 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_duration": 0, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_trimmed": 0, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrim_duration": 0, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "stat_sum": { 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes": 413696, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects": 101, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_clones": 0, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_copies": 505, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing_on_primary": 1, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing": 0, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_degraded": 3, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_misplaced": 0, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_unfound": 1, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_dirty": 101, 2024-03-25T00:47:43.455 INFO:tasks.workunit.client.0.smithi080.stdout: "num_whiteouts": 0, 2024-03-25T00:47:43.456 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read": 0, 2024-03-25T00:47:43.456 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read_kb": 0, 2024-03-25T00:47:43.456 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write": 101, 2024-03-25T00:47:43.456 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write_kb": 404, 2024-03-25T00:47:43.456 INFO:tasks.workunit.client.0.smithi080.stdout: "num_scrub_errors": 0, 2024-03-25T00:47:43.456 INFO:tasks.workunit.client.0.smithi080.stdout: "num_shallow_scrub_errors": 0, 2024-03-25T00:47:43.456 INFO:tasks.workunit.client.0.smithi080.stdout: "num_deep_scrub_errors": 0, 2024-03-25T00:47:43.456 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_recovered": 99, 2024-03-25T00:47:43.456 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_recovered": 405504, 2024-03-25T00:47:43.456 INFO:tasks.workunit.client.0.smithi080.stdout: "num_keys_recovered": 0, 2024-03-25T00:47:43.456 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_omap": 0, 2024-03-25T00:47:43.457 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_hit_set_archive": 0, 2024-03-25T00:47:43.457 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_hit_set_archive": 0, 2024-03-25T00:47:43.457 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush": 0, 2024-03-25T00:47:43.457 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_kb": 0, 2024-03-25T00:47:43.457 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict": 0, 2024-03-25T00:47:43.457 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_kb": 0, 2024-03-25T00:47:43.457 INFO:tasks.workunit.client.0.smithi080.stdout: "num_promote": 0, 2024-03-25T00:47:43.457 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_high": 0, 2024-03-25T00:47:43.457 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_low": 0, 2024-03-25T00:47:43.457 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_some": 0, 2024-03-25T00:47:43.458 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_full": 0, 2024-03-25T00:47:43.458 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_pinned": 0, 2024-03-25T00:47:43.458 INFO:tasks.workunit.client.0.smithi080.stdout: "num_legacy_snapsets": 0, 2024-03-25T00:47:43.458 INFO:tasks.workunit.client.0.smithi080.stdout: "num_large_omap_objects": 0, 2024-03-25T00:47:43.458 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_manifest": 0, 2024-03-25T00:47:43.458 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_bytes": 0, 2024-03-25T00:47:43.458 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_keys": 0, 2024-03-25T00:47:43.458 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_repaired": 0 2024-03-25T00:47:43.458 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.459 INFO:tasks.workunit.client.0.smithi080.stdout: "up": [ 2024-03-25T00:47:43.459 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.459 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.459 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.459 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.459 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.459 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.459 INFO:tasks.workunit.client.0.smithi080.stdout: "acting": [ 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: "avail_no_missing": [ 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: "object_location_counts": [ 2024-03-25T00:47:43.460 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": 100 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: "shards": "0(3),4(2)", 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": 1 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: "blocked_by": [], 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: "up_primary": 3, 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: "acting_primary": 3, 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [] 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.461 INFO:tasks.workunit.client.0.smithi080.stdout: "empty": 0, 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: "dne": 0, 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: "incomplete": 0, 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 56, 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: "hit_set_history": { 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: "current_last_update": "0'0", 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: "history": [] 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: "peer_info": [ 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: "peer": "0(3)", 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: "pgid": "2.0s3", 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: "shared": "3", 2024-03-25T00:47:43.462 INFO:tasks.workunit.client.0.smithi080.stdout: "last_update": "52'101", 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "last_complete": "52'101", 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "log_tail": "0'0", 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "last_user_version": 101, 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "last_backfill": "MAX", 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [], 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "history": { 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_created": 44, 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_pool_created": 44, 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 56, 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_started": 55, 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 45, 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_clean": 44, 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_split": 0, 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_marked_full": 0, 2024-03-25T00:47:43.463 INFO:tasks.workunit.client.0.smithi080.stdout: "same_up_since": 55, 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "same_interval_since": 55, 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "same_primary_since": 44, 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "prior_readable_until_ub": 0 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "stats": { 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "version": "52'101", 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_seq": 133, 2024-03-25T00:47:43.464 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_epoch": 52, 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "active+undersized+degraded", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fresh": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "last_change": "2024-03-25T00:46:53.020960+0000", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "last_active": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "last_peered": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean": "2024-03-25T00:46:51.622310+0000", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_active": "2024-03-25T00:46:53.020960+0000", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_peered": "2024-03-25T00:46:53.020960+0000", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "last_unstale": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "last_undegraded": "2024-03-25T00:46:52.759913+0000", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fullsized": "2024-03-25T00:46:52.759696+0000", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "mapping_epoch": 55, 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "log_start": "0'0", 2024-03-25T00:47:43.465 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_start": "0'0", 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "created": 44, 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 45, 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "parent": "0.0", 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "parent_split_bits": 0, 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_scrubbed": 0, 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "log_size": 101, 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "log_dups_size": 0, 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_size": 101, 2024-03-25T00:47:43.466 INFO:tasks.workunit.client.0.smithi080.stdout: "stats_invalid": false, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "dirty_stats_invalid": false, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "omap_stats_invalid": false, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_stats_invalid": false, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_bytes_stats_invalid": false, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "pin_stats_invalid": false, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "manifest_stats_invalid": false, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrimq_len": 0, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_duration": 0, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-03-26T06:20:24.974329+0000", 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_duration": 0, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_trimmed": 0, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrim_duration": 0, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "stat_sum": { 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes": 413696, 2024-03-25T00:47:43.467 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects": 101, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_clones": 0, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_copies": 505, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing_on_primary": 0, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing": 0, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_degraded": 101, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_misplaced": 0, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_unfound": 0, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_dirty": 101, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_whiteouts": 0, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read": 0, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read_kb": 0, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write": 101, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write_kb": 404, 2024-03-25T00:47:43.468 INFO:tasks.workunit.client.0.smithi080.stdout: "num_scrub_errors": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_shallow_scrub_errors": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_deep_scrub_errors": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_recovered": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_recovered": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_keys_recovered": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_omap": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_hit_set_archive": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_hit_set_archive": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_kb": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_kb": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_promote": 0, 2024-03-25T00:47:43.469 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_high": 0, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_low": 0, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_some": 0, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_full": 0, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_pinned": 0, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: "num_legacy_snapsets": 0, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: "num_large_omap_objects": 0, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_manifest": 0, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_bytes": 0, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_keys": 0, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_repaired": 0 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: "up": [ 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.470 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: "acting": [ 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: "avail_no_missing": [ 2024-03-25T00:47:43.471 INFO:tasks.workunit.client.0.smithi080.stdout: "3(0)", 2024-03-25T00:47:43.472 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:47:43.472 INFO:tasks.workunit.client.0.smithi080.stdout: "1(1)", 2024-03-25T00:47:43.472 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:47:43.472 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.472 INFO:tasks.workunit.client.0.smithi080.stdout: "object_location_counts": [ 2024-03-25T00:47:43.472 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.472 INFO:tasks.workunit.client.0.smithi080.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-03-25T00:47:43.472 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": 101 2024-03-25T00:47:43.472 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.472 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: "blocked_by": [], 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: "up_primary": 3, 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: "acting_primary": 3, 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [] 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: "empty": 0, 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: "dne": 0, 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: "incomplete": 0, 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 56, 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: "hit_set_history": { 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: "current_last_update": "0'0", 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: "history": [] 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.473 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "peer": "1(1)", 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "pgid": "2.0s1", 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "shared": "1", 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "last_update": "52'101", 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "last_complete": "52'101", 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "log_tail": "0'0", 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "last_user_version": 101, 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "last_backfill": "MAX", 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [], 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "history": { 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_created": 44, 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_pool_created": 44, 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 56, 2024-03-25T00:47:43.474 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_started": 55, 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 45, 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_clean": 44, 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_split": 0, 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_marked_full": 0, 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "same_up_since": 55, 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "same_interval_since": 55, 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "same_primary_since": 44, 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "prior_readable_until_ub": 0 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.475 INFO:tasks.workunit.client.0.smithi080.stdout: "stats": { 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "version": "52'101", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_seq": 133, 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_epoch": 52, 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "active+undersized+degraded", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fresh": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "last_change": "2024-03-25T00:46:53.020960+0000", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "last_active": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "last_peered": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean": "2024-03-25T00:46:51.622310+0000", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_active": "2024-03-25T00:46:53.020960+0000", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_peered": "2024-03-25T00:46:53.020960+0000", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "last_unstale": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "last_undegraded": "2024-03-25T00:46:52.759913+0000", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fullsized": "2024-03-25T00:46:52.759696+0000", 2024-03-25T00:47:43.476 INFO:tasks.workunit.client.0.smithi080.stdout: "mapping_epoch": 55, 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "log_start": "0'0", 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_start": "0'0", 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "created": 44, 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 45, 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "parent": "0.0", 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "parent_split_bits": 0, 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_scrubbed": 0, 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "log_size": 101, 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "log_dups_size": 0, 2024-03-25T00:47:43.477 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_size": 101, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "stats_invalid": false, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "dirty_stats_invalid": false, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "omap_stats_invalid": false, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_stats_invalid": false, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_bytes_stats_invalid": false, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "pin_stats_invalid": false, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "manifest_stats_invalid": false, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrimq_len": 0, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_duration": 0, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-03-26T06:20:24.974329+0000", 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_duration": 0, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_trimmed": 0, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrim_duration": 0, 2024-03-25T00:47:43.478 INFO:tasks.workunit.client.0.smithi080.stdout: "stat_sum": { 2024-03-25T00:47:43.479 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes": 413696, 2024-03-25T00:47:43.479 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects": 101, 2024-03-25T00:47:43.479 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_clones": 0, 2024-03-25T00:47:43.479 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_copies": 505, 2024-03-25T00:47:43.479 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing_on_primary": 0, 2024-03-25T00:47:43.481 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing": 1, 2024-03-25T00:47:43.481 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_degraded": 101, 2024-03-25T00:47:43.481 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_misplaced": 0, 2024-03-25T00:47:43.481 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_unfound": 0, 2024-03-25T00:47:43.481 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_dirty": 101, 2024-03-25T00:47:43.481 INFO:tasks.workunit.client.0.smithi080.stdout: "num_whiteouts": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read_kb": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write": 101, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write_kb": 404, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_scrub_errors": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_shallow_scrub_errors": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_deep_scrub_errors": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_recovered": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_recovered": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_keys_recovered": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_omap": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_hit_set_archive": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_hit_set_archive": 0, 2024-03-25T00:47:43.482 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_kb": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_kb": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_promote": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_high": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_low": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_some": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_full": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_pinned": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_legacy_snapsets": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_large_omap_objects": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_manifest": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_bytes": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_keys": 0, 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_repaired": 0 2024-03-25T00:47:43.483 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: "up": [ 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: "acting": [ 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.484 INFO:tasks.workunit.client.0.smithi080.stdout: "avail_no_missing": [ 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: "3(0)", 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: "1(1)", 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: "object_location_counts": [ 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": 101 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: "blocked_by": [], 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: "up_primary": 3, 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: "acting_primary": 3, 2024-03-25T00:47:43.485 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [] 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: "empty": 0, 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: "dne": 0, 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: "incomplete": 0, 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 56, 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: "hit_set_history": { 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: "current_last_update": "0'0", 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: "history": [] 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: "peer": "2(4)", 2024-03-25T00:47:43.486 INFO:tasks.workunit.client.0.smithi080.stdout: "pgid": "2.0s4", 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "shared": "4", 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "last_update": "52'101", 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "last_complete": "46'1", 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "log_tail": "0'0", 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "last_user_version": 1, 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "last_backfill": "MAX", 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [], 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "history": { 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_created": 44, 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_pool_created": 44, 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 56, 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_started": 55, 2024-03-25T00:47:43.487 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 45, 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_clean": 44, 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_split": 0, 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_marked_full": 0, 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "same_up_since": 55, 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "same_interval_since": 55, 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "same_primary_since": 44, 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "prior_readable_until_ub": 0 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "stats": { 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "version": "46'1", 2024-03-25T00:47:43.488 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_seq": 17, 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_epoch": 46, 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "active+clean", 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fresh": "2024-03-25T00:46:51.622310+0000", 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "last_change": "2024-03-25T00:46:44.884719+0000", 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "last_active": "2024-03-25T00:46:51.622310+0000", 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "last_peered": "2024-03-25T00:46:51.622310+0000", 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean": "2024-03-25T00:46:51.622310+0000", 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_active": "2024-03-25T00:46:44.884145+0000", 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_peered": "2024-03-25T00:46:44.884145+0000", 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "last_unstale": "2024-03-25T00:46:51.622310+0000", 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "last_undegraded": "2024-03-25T00:46:51.622310+0000", 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fullsized": "2024-03-25T00:46:51.622310+0000", 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "mapping_epoch": 55, 2024-03-25T00:47:43.489 INFO:tasks.workunit.client.0.smithi080.stdout: "log_start": "0'0", 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_start": "0'0", 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "created": 44, 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 45, 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "parent": "0.0", 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "parent_split_bits": 0, 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_scrubbed": 0, 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "log_size": 1, 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "log_dups_size": 0, 2024-03-25T00:47:43.490 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_size": 1, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "stats_invalid": false, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "dirty_stats_invalid": false, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "omap_stats_invalid": false, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_stats_invalid": false, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_bytes_stats_invalid": false, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "pin_stats_invalid": false, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "manifest_stats_invalid": false, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrimq_len": 0, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_duration": 0, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-03-26T02:09:41.300317+0000", 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_duration": 0, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_trimmed": 0, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrim_duration": 0, 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "stat_sum": { 2024-03-25T00:47:43.491 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes": 4096, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects": 1, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_clones": 0, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_copies": 5, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing_on_primary": 0, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing": 1, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_degraded": 0, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_misplaced": 0, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_unfound": 0, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_dirty": 1, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_whiteouts": 0, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read": 0, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read_kb": 0, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write": 1, 2024-03-25T00:47:43.492 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write_kb": 4, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_scrub_errors": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_shallow_scrub_errors": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_deep_scrub_errors": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_recovered": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_recovered": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_keys_recovered": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_omap": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_hit_set_archive": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_hit_set_archive": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_kb": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_kb": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_promote": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_high": 0, 2024-03-25T00:47:43.493 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_low": 0, 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_some": 0, 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_full": 0, 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_pinned": 0, 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: "num_legacy_snapsets": 0, 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: "num_large_omap_objects": 0, 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_manifest": 0, 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_bytes": 0, 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_keys": 0, 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_repaired": 0 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: "up": [ 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.494 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: "acting": [ 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: "avail_no_missing": [], 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: "object_location_counts": [], 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: "blocked_by": [], 2024-03-25T00:47:43.495 INFO:tasks.workunit.client.0.smithi080.stdout: "up_primary": 3, 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: "acting_primary": 3, 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [] 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: "empty": 0, 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: "dne": 0, 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: "incomplete": 0, 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 56, 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: "hit_set_history": { 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: "current_last_update": "0'0", 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: "history": [] 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.496 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.497 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.497 INFO:tasks.workunit.client.0.smithi080.stdout: "peer": "4(2)", 2024-03-25T00:47:43.497 INFO:tasks.workunit.client.0.smithi080.stdout: "pgid": "2.0s2", 2024-03-25T00:47:43.497 INFO:tasks.workunit.client.0.smithi080.stdout: "shared": "2", 2024-03-25T00:47:43.497 INFO:tasks.workunit.client.0.smithi080.stdout: "last_update": "52'101", 2024-03-25T00:47:43.497 INFO:tasks.workunit.client.0.smithi080.stdout: "last_complete": "52'101", 2024-03-25T00:47:43.497 INFO:tasks.workunit.client.0.smithi080.stdout: "log_tail": "0'0", 2024-03-25T00:47:43.497 INFO:tasks.workunit.client.0.smithi080.stdout: "last_user_version": 101, 2024-03-25T00:47:43.497 INFO:tasks.workunit.client.0.smithi080.stdout: "last_backfill": "MAX", 2024-03-25T00:47:43.497 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [], 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "history": { 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_created": 44, 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "epoch_pool_created": 44, 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 56, 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_started": 55, 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 45, 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "last_interval_clean": 44, 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_split": 0, 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_marked_full": 0, 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "same_up_since": 55, 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "same_interval_since": 55, 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "same_primary_since": 44, 2024-03-25T00:47:43.498 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "prior_readable_until_ub": 0 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "stats": { 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "version": "52'101", 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_seq": 133, 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "reported_epoch": 52, 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "state": "active+undersized+degraded", 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fresh": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "last_change": "2024-03-25T00:46:53.020960+0000", 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "last_active": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.499 INFO:tasks.workunit.client.0.smithi080.stdout: "last_peered": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean": "2024-03-25T00:46:51.622310+0000", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_active": "2024-03-25T00:46:53.020960+0000", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "last_became_peered": "2024-03-25T00:46:53.020960+0000", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "last_unstale": "2024-03-25T00:47:20.249241+0000", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "last_undegraded": "2024-03-25T00:46:52.759913+0000", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "last_fullsized": "2024-03-25T00:46:52.759696+0000", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "mapping_epoch": 55, 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "log_start": "0'0", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_start": "0'0", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "created": 44, 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_clean": 45, 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "parent": "0.0", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "parent_split_bits": 0, 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub": "0'0", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.500 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub": "0'0", 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "last_deep_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "last_clean_scrub_stamp": "2024-03-25T00:46:42.990182+0000", 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_scrubbed": 0, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "log_size": 101, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "log_dups_size": 0, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "ondisk_log_size": 101, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "stats_invalid": false, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "dirty_stats_invalid": false, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "omap_stats_invalid": false, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_stats_invalid": false, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "hitset_bytes_stats_invalid": false, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "pin_stats_invalid": false, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "manifest_stats_invalid": false, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrimq_len": 0, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "last_scrub_duration": 0, 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-03-26T06:20:24.974329+0000", 2024-03-25T00:47:43.501 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_duration": 0, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "objects_trimmed": 0, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "snaptrim_duration": 0, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "stat_sum": { 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes": 413696, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects": 101, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_clones": 0, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "num_object_copies": 505, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing_on_primary": 0, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_missing": 0, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_degraded": 101, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_misplaced": 0, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_unfound": 0, 2024-03-25T00:47:43.502 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_dirty": 101, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_whiteouts": 0, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read": 0, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_read_kb": 0, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write": 101, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_write_kb": 404, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_scrub_errors": 0, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_shallow_scrub_errors": 0, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_deep_scrub_errors": 0, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_recovered": 0, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_recovered": 0, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_keys_recovered": 0, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_omap": 0, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_hit_set_archive": 0, 2024-03-25T00:47:43.503 INFO:tasks.workunit.client.0.smithi080.stdout: "num_bytes_hit_set_archive": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_kb": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_kb": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_promote": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_high": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_flush_mode_low": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_some": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_evict_mode_full": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_pinned": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_legacy_snapsets": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_large_omap_objects": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_manifest": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_bytes": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_omap_keys": 0, 2024-03-25T00:47:43.504 INFO:tasks.workunit.client.0.smithi080.stdout: "num_objects_repaired": 0 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: "up": [ 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: "acting": [ 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: 3, 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: 1, 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: 4, 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: 0, 2024-03-25T00:47:43.505 INFO:tasks.workunit.client.0.smithi080.stdout: 2 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: "avail_no_missing": [ 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: "3(0)", 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: "0(3)", 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: "1(1)", 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: "4(2)" 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: "object_location_counts": [ 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": 101 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.506 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: "blocked_by": [], 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: "up_primary": 3, 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: "acting_primary": 3, 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: "purged_snaps": [] 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: "empty": 0, 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: "dne": 0, 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: "incomplete": 0, 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: "last_epoch_started": 56, 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: "hit_set_history": { 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: "current_last_update": "0'0", 2024-03-25T00:47:43.507 INFO:tasks.workunit.client.0.smithi080.stdout: "history": [] 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: "recovery_state": [ 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: "name": "Started/Primary/Active", 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: "enter_time": "2024-03-25T00:47:27.927586+0000", 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: "might_have_unfound": [ 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: "osd": "0(3)", 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: "status": "already probed" 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.508 INFO:tasks.workunit.client.0.smithi080.stdout: "osd": "1(1)", 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: "status": "already probed" 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: "osd": "2(4)", 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: "status": "already probed" 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: "osd": "4(2)", 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: "status": "already probed" 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: "recovery_progress": { 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: "backfill_targets": [], 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: "waiting_on_backfill": [], 2024-03-25T00:47:43.509 INFO:tasks.workunit.client.0.smithi080.stdout: "last_backfill_started": "MIN", 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: "backfill_info": { 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: "begin": "MIN", 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: "end": "MIN", 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: "objects": [] 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: "peer_backfill_info": [], 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: "backfills_in_flight": [], 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: "recovering": [], 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: "pg_backend": { 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: "recovery_ops": [], 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: "read_ops": [] 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.510 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: { 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: "name": "Started", 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: "enter_time": "2024-03-25T00:47:27.458702+0000" 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: } 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: ], 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: "scrubber": { 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: "active": false, 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: "must_scrub": false, 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: "must_deep_scrub": false, 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: "must_repair": false, 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: "need_auto": false, 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: "scrub_reg_stamp": "2024-03-26T06:20:24.974329+0000", 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: "schedule": "no scrub is scheduled" 2024-03-25T00:47:43.511 INFO:tasks.workunit.client.0.smithi080.stdout: }, 2024-03-25T00:47:43.512 INFO:tasks.workunit.client.0.smithi080.stdout: "agent_state": {} 2024-03-25T00:47:43.512 INFO:tasks.workunit.client.0.smithi080.stdout:} 2024-03-25T00:47:43.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-03-25T00:47:43.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: grep -q obj75 2024-03-25T00:47:43.637 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-03-25T00:47:43.638 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2024-03-25T00:47:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: check=true 2024-03-25T00:47:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:667: TEST_ec_recovery_unfound: test true == true 2024-03-25T00:47:43.796 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-03-25T00:47:43.796 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2024-03-25T00:47:43.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: check=0 2024-03-25T00:47:43.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: test 0 == 0 2024-03-25T00:47:43.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-03-25T00:47:48.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: test 124 = 124 2024-03-25T00:47:48.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:676: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-03-25T00:47:49.250 INFO:tasks.workunit.client.0.smithi080.stderr:pg has no unfound objects 2024-03-25T00:47:49.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:678: TEST_ec_recovery_unfound: wait_for_clean 2024-03-25T00:47:49.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:47:49.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:47:49.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:47:49.269 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:47:49.269 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:47:49.270 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:47:49.270 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:47:49.270 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:47:49.270 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:47:49.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:47:49.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:47:49.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:47:49.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:47:49.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:47:49.389 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:47:49.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:47:49.903 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:47:49.903 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:47:49.903 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:47:49.903 INFO:tasks.workunit.client.0.smithi080.stderr:4' 2024-03-25T00:47:49.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:47:49.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:47:49.904 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:47:50.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803825 2024-03-25T00:47:50.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803825 2024-03-25T00:47:50.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825' 2024-03-25T00:47:50.068 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:47:50.068 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:47:50.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574894 2024-03-25T00:47:50.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574894 2024-03-25T00:47:50.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825 1-55834574894' 2024-03-25T00:47:50.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:47:50.244 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:47:50.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233992 2024-03-25T00:47:50.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233992 2024-03-25T00:47:50.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825 1-55834574894 2-231928233992' 2024-03-25T00:47:50.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:47:50.396 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:47:50.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117022 2024-03-25T00:47:50.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117022 2024-03-25T00:47:50.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825 1-55834574894 2-231928233992 3-115964117022' 2024-03-25T00:47:50.548 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:47:50.548 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:47:50.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757271 2024-03-25T00:47:50.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757271 2024-03-25T00:47:50.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803825 1-55834574894 2-231928233992 3-115964117022 4-163208757271' 2024-03-25T00:47:50.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:47:50.699 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803825 2024-03-25T00:47:50.699 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:47:50.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:47:50.700 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803825 2024-03-25T00:47:50.701 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:47:50.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803825 2024-03-25T00:47:50.702 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803825 2024-03-25T00:47:50.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803825' 2024-03-25T00:47:50.702 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:47:51.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803825 -lt 25769803825 2024-03-25T00:47:51.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:47:51.116 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574894 2024-03-25T00:47:51.116 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:47:51.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:47:51.118 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574894 2024-03-25T00:47:51.119 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:47:51.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574894 2024-03-25T00:47:51.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574894' 2024-03-25T00:47:51.120 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 55834574894 2024-03-25T00:47:51.121 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:47:51.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574894 -lt 55834574894 2024-03-25T00:47:51.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:47:51.569 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-231928233992 2024-03-25T00:47:51.569 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:47:51.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:47:51.572 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-231928233992 2024-03-25T00:47:51.572 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:47:51.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233992 2024-03-25T00:47:51.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 231928233992' 2024-03-25T00:47:51.574 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 231928233992 2024-03-25T00:47:51.574 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:47:51.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233992 -lt 231928233992 2024-03-25T00:47:51.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:47:51.990 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117022 2024-03-25T00:47:51.990 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:47:51.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:47:51.992 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117022 2024-03-25T00:47:51.992 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:47:51.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117022 2024-03-25T00:47:51.994 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 115964117022 2024-03-25T00:47:51.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117022' 2024-03-25T00:47:51.994 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:47:52.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117022 -lt 115964117022 2024-03-25T00:47:52.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:47:52.406 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757271 2024-03-25T00:47:52.406 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:47:52.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:47:52.408 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757271 2024-03-25T00:47:52.408 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:47:52.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757271 2024-03-25T00:47:52.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757271' 2024-03-25T00:47:52.410 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 163208757271 2024-03-25T00:47:52.410 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:47:52.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757271 -lt 163208757271 2024-03-25T00:47:52.867 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:47:52.867 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:47:52.867 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:47:53.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:47:53.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:47:53.411 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:47:53.411 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:47:53.411 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:47:53.411 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:47:53.411 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:47:53.411 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:47:53.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:47:53.820 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:47:53.820 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:47:53.820 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:47:54.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:47:54.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:47:54.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:47:54.345 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: seq 1 100 2024-03-25T00:47:54.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2024-03-25T00:47:54.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2024-03-25T00:47:54.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2024-03-25T00:47:54.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2024-03-25T00:47:54.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2024-03-25T00:47:54.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2024-03-25T00:47:54.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2024-03-25T00:47:54.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2024-03-25T00:47:54.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2024-03-25T00:47:54.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2024-03-25T00:47:54.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2024-03-25T00:47:54.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.767 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2024-03-25T00:47:54.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2024-03-25T00:47:54.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2024-03-25T00:47:54.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2024-03-25T00:47:54.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2024-03-25T00:47:54.883 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2024-03-25T00:47:54.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2024-03-25T00:47:54.941 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.969 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2024-03-25T00:47:54.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-03-25T00:47:54.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:54.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:54.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2024-03-25T00:47:54.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2024-03-25T00:47:55.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2024-03-25T00:47:55.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.106 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2024-03-25T00:47:55.107 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.135 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2024-03-25T00:47:55.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2024-03-25T00:47:55.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.194 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2024-03-25T00:47:55.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2024-03-25T00:47:55.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2024-03-25T00:47:55.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2024-03-25T00:47:55.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2024-03-25T00:47:55.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2024-03-25T00:47:55.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2024-03-25T00:47:55.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2024-03-25T00:47:55.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.423 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2024-03-25T00:47:55.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2024-03-25T00:47:55.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2024-03-25T00:47:55.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2024-03-25T00:47:55.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2024-03-25T00:47:55.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2024-03-25T00:47:55.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.628 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2024-03-25T00:47:55.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.658 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.658 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2024-03-25T00:47:55.658 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2024-03-25T00:47:55.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.715 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2024-03-25T00:47:55.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2024-03-25T00:47:55.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2024-03-25T00:47:55.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2024-03-25T00:47:55.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2024-03-25T00:47:55.832 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2024-03-25T00:47:55.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2024-03-25T00:47:55.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2024-03-25T00:47:55.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2024-03-25T00:47:55.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-03-25T00:47:55.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:55.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:55.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2024-03-25T00:47:55.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2024-03-25T00:47:56.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2024-03-25T00:47:56.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.064 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2024-03-25T00:47:56.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2024-03-25T00:47:56.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2024-03-25T00:47:56.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2024-03-25T00:47:56.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2024-03-25T00:47:56.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2024-03-25T00:47:56.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2024-03-25T00:47:56.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2024-03-25T00:47:56.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2024-03-25T00:47:56.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2024-03-25T00:47:56.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2024-03-25T00:47:56.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2024-03-25T00:47:56.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2024-03-25T00:47:56.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2024-03-25T00:47:56.441 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2024-03-25T00:47:56.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2024-03-25T00:47:56.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2024-03-25T00:47:56.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2024-03-25T00:47:56.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2024-03-25T00:47:56.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2024-03-25T00:47:56.612 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.641 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2024-03-25T00:47:56.641 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.666 INFO:tasks.workunit.client.0.smithi080.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2024-03-25T00:47:56.668 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.668 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2024-03-25T00:47:56.668 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2024-03-25T00:47:56.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.725 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2024-03-25T00:47:56.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2024-03-25T00:47:56.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2024-03-25T00:47:56.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2024-03-25T00:47:56.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2024-03-25T00:47:56.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.868 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2024-03-25T00:47:56.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.897 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2024-03-25T00:47:56.898 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2024-03-25T00:47:56.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2024-03-25T00:47:56.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-03-25T00:47:56.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:56.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:56.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2024-03-25T00:47:56.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2024-03-25T00:47:57.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.060 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2024-03-25T00:47:57.061 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2024-03-25T00:47:57.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2024-03-25T00:47:57.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2024-03-25T00:47:57.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2024-03-25T00:47:57.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2024-03-25T00:47:57.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2024-03-25T00:47:57.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2024-03-25T00:47:57.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2024-03-25T00:47:57.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2024-03-25T00:47:57.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2024-03-25T00:47:57.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-03-25T00:47:57.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2024-03-25T00:47:57.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-03-25T00:47:57.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:691: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-03-25T00:47:57.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2024-03-25T00:47:57.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:47:57.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-03-25T00:47:57.902 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' does not exist 2024-03-25T00:47:57.918 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-03-25T00:47:58.695 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile myprofile does not exist 2024-03-25T00:47:58.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-03-25T00:47:58.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:47:58.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:47:58.713 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:47:58.713 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:47:58.714 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:47:58.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:47:58.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:47:58.714 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:47:59.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:47:59.028 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:47:59.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:47:59.029 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:47:59.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:47:59.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:47:59.032 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:47:59.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:47:59.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:47:59.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:47:59.034 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:47:59.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:47:59.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:47:59.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:47:59.076 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:47:59.077 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:47:59.077 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:47:59.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:47:59.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:47:59.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:47:59.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-03-25T00:47:59.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-03-25T00:47:59.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-03-25T00:47:59.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-03-25T00:47:59.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:47:59.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:47:59.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:47:59.079 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:47:59.080 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:47:59.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:47:59.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:47:59.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:47:59.082 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:47:59.082 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:47:59.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:47:59.083 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:47:59.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:47:59.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:47:59.085 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:47:59.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:47:59.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:47:59.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:47:59.087 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:47:59.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:47:59.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:47:59.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:47:59.090 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:47:59.090 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:47:59.090 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:47:59.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:47:59.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:47:59.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:47:59.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-03-25T00:47:59.093 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:47:59.093 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:47:59.093 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:47:59.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81310 2024-03-25T00:47:59.094 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:47:59.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:47:59.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:47:59.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-03-25T00:47:59.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-03-25T00:47:59.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-03-25T00:47:59.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:47:59.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:47:59.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:47:59.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-03-25T00:47:59.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-03-25T00:47:59.589 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:47:59.589 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:47:59.589 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:47:59.590 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:47:59.590 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:47:59.590 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:47:59.590 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:47:59.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T00:47:59.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:47:59.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:47:59.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:47:59.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:47:59.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:47:59.783 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:47:59.784 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:47:59.784 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:47:59.784 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:47:59.784 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:47:59.785 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:47:59.785 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:47:59.785 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:47:59.785 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:47:59.785 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get fsid 2024-03-25T00:47:59.905 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:47:59.905 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:47:59.905 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:47:59.905 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:47:59.906 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:47:59.906 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:47:59.906 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:47:59.906 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:47:59.907 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:47:59.907 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:47:59.907 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:47:59.907 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:47:59.908 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:47:59.908 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get mon_host 2024-03-25T00:48:00.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-03-25T00:48:00.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-03-25T00:48:00.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:48:00.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:48:00.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:48:00.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-03-25T00:48:00.014 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:48:00.899 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:48:00.899 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:48:00.899 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:48:00.899 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:48:00.899 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:48:00.899 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:48:00.899 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:48:00.899 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:48:00.900 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:48:00.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-03-25T00:48:00.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T00:48:01.221 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' already exists 2024-03-25T00:48:01.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:48:02.240 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-03-25T00:48:02.240 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:48:02.240 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:48:02.241 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:48:02.241 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:48:02.241 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:48:02.241 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:48:02.241 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-03-25T00:48:02.241 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-mon.a.asok log flush 2024-03-25T00:48:02.348 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:48:02.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-03-25T00:48:02.363 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:47:59.631+0000 7f47401cfc80 0 load: jerasure load: lrc 2024-03-25T00:48:02.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2024-03-25T00:48:02.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:390: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2024-03-25T00:48:02.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:391: TEST_ec_single_recovery_error: local objname=myobject 2024-03-25T00:48:02.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:393: TEST_ec_single_recovery_error: setup_osds 7 2024-03-25T00:48:02.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-03-25T00:48:02.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-03-25T00:48:02.365 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-03-25T00:48:02.366 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-03-25T00:48:02.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:48:02.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-03-25T00:48:02.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:48:02.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:48:02.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:48:02.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:48:02.368 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-03-25T00:48:02.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:48:02.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:48:02.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:48:02.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:48:02.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-03-25T00:48:02.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-03-25T00:48:02.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:48:02.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:48:02.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:48:02.371 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:48:02.371 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:48:02.371 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:48:02.371 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:48:02.371 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:48:02.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:48:02.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:48:02.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-03-25T00:48:02.373 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:48:02.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=55df4884-ac75-4d99-8680-28544247bde9 2024-03-25T00:48:02.373 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 55df4884-ac75-4d99-8680-28544247bde9 2024-03-25T00:48:02.374 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 55df4884-ac75-4d99-8680-28544247bde9' 2024-03-25T00:48:02.374 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:48:02.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCyQBmRziVFxAAjqfElH6hK6bvm49AmmoM6Q== 2024-03-25T00:48:02.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCyQBmRziVFxAAjqfElH6hK6bvm49AmmoM6Q=="}' 2024-03-25T00:48:02.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 55df4884-ac75-4d99-8680-28544247bde9 -i td/test-erasure-eio/0/new.json 2024-03-25T00:48:02.714 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:48:02.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-03-25T00:48:02.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCyQBmRziVFxAAjqfElH6hK6bvm49AmmoM6Q== --osd-uuid 55df4884-ac75-4d99-8680-28544247bde9 2024-03-25T00:48:02.772 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:02.771+0000 7fd60f9a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:02.799 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:02.795+0000 7fd60f9a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:02.800 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:02.799+0000 7fd60f9a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:02.801 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:02.799+0000 7fd60f9a07c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:48:02.801 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:02.799+0000 7fd60f9a07c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:48:02.801 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:02.799+0000 7fd60f9a07c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:48:02.839 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:02.835+0000 7fd60f9a07c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-03-25T00:48:08.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-03-25T00:48:08.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:48:08.908 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:48:08.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:48:08.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:48:09.459 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:48:09.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:48:09.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:48:09.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:48:09.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:48:09.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:48:09.491 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:09.486+0000 7fe0d5aa47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:09.493 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:09.490+0000 7fe0d5aa47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:09.494 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:09.490+0000 7fe0d5aa47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:09.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:48:09.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:48:09.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:48:09.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:48:09.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:48:09.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:09.879 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:48:09.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:48:09.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:09.880 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:10.078 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:10.074+0000 7fe0d5aa47c0 -1 Falling back to public interface 2024-03-25T00:48:10.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:11.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:11.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:11.303 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:48:11.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:48:11.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:11.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:11.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:12.633 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:12.630+0000 7fe0d5aa47c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:48:12.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:12.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:12.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:48:12.702 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:48:12.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:12.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:13.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:14.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:14.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:14.308 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:48:14.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:48:14.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:14.309 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:14.767 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:15.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:15.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:15.769 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:48:15.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:48:15.770 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:15.770 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:16.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:17.232 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:17.232 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:17.232 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:48:17.233 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:48:17.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:17.233 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:17.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:18.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:18.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:18.740 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:48:18.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:48:18.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:18.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:19.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:20.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:20.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:20.177 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:48:20.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:48:20.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:20.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:20.800 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:21.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:21.802 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:21.802 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:48:21.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:48:21.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:21.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:22.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:23.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:23.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:23.248 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:48:23.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:48:23.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:23.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:23.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:24.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:24.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:24.736 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:48:24.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:48:24.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:24.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:25.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:26.200 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:26.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:26.201 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:48:26.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:48:26.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:26.202 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:27.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:28.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:28.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:28.072 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:48:28.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:48:28.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:28.073 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:28.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:29.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:29.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:29.948 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:48:29.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:48:29.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:29.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:30.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:31.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:31.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:31.997 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:48:31.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:48:31.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:31.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:32.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:33.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:33.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:33.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:48:33.914 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:48:33.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:33.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:34.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:35.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:35.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:35.854 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:48:35.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:48:35.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:35.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:48:36.359 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1933165790,v1:127.0.0.1:6803/1933165790] [v2:127.0.0.1:6804/1933165790,v1:127.0.0.1:6805/1933165790] exists,up 55df4884-ac75-4d99-8680-28544247bde9 2024-03-25T00:48:36.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:48:36.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:48:36.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:48:36.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:48:36.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-03-25T00:48:36.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:48:36.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:48:36.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T00:48:36.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:48:36.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:48:36.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:48:36.364 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:48:36.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:48:36.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:48:36.366 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:48:36.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2ef313ef-752e-46e4-89f8-c3fd253ed8ff 2024-03-25T00:48:36.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2ef313ef-752e-46e4-89f8-c3fd253ed8ff' 2024-03-25T00:48:36.367 INFO:tasks.workunit.client.0.smithi080.stdout:add osd1 2ef313ef-752e-46e4-89f8-c3fd253ed8ff 2024-03-25T00:48:36.367 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:48:36.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkyQBmPlgCFxAA+hQu/EKYtBSiEZfYER0tuQ== 2024-03-25T00:48:36.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkyQBmPlgCFxAA+hQu/EKYtBSiEZfYER0tuQ=="}' 2024-03-25T00:48:36.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2ef313ef-752e-46e4-89f8-c3fd253ed8ff -i td/test-erasure-eio/1/new.json 2024-03-25T00:48:36.870 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:48:36.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-03-25T00:48:36.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkyQBmPlgCFxAA+hQu/EKYtBSiEZfYER0tuQ== --osd-uuid 2ef313ef-752e-46e4-89f8-c3fd253ed8ff 2024-03-25T00:48:36.921 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:36.918+0000 7f9a40f0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:36.923 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:36.918+0000 7f9a40f0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:36.924 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:36.922+0000 7f9a40f0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:36.924 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:36.922+0000 7f9a40f0c7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:48:36.924 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:36.922+0000 7f9a40f0c7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:48:36.924 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:36.922+0000 7f9a40f0c7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:48:36.987 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:36.982+0000 7f9a40f0c7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-03-25T00:48:43.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-03-25T00:48:43.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:48:43.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T00:48:43.572 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd1 key to auth repository 2024-03-25T00:48:43.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:48:44.117 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.1 2024-03-25T00:48:44.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T00:48:44.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:48:44.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:48:44.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:48:44.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:48:44.148 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:44.146+0000 7fc26a9367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:44.169 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:44.166+0000 7fc26a9367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:44.170 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:44.166+0000 7fc26a9367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:48:44.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T00:48:44.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:48:44.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:48:44.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:48:44.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:48:44.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:44.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:48:44.547 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:48:44.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:44.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:48:44.749 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:44.746+0000 7fc26a9367c0 -1 Falling back to public interface 2024-03-25T00:48:44.970 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:45.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:45.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:45.972 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:48:45.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:48:45.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:45.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:48:46.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:47.265 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:48:47.262+0000 7fc26a9367c0 -1 osd.1 0 log_to_monitors true 2024-03-25T00:48:47.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:47.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:47.386 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:48:47.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:48:47.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:47.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:48:47.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:48.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:48.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:48.994 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:48:48.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:48:48.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:48.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:48:49.442 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:50.443 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:50.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:50.444 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:48:50.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:48:50.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:50.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:48:50.914 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:51.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:51.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:51.917 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:48:51.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:48:51.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:51.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:48:52.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:53.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:53.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:53.570 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:48:53.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:48:53.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:53.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:48:54.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:55.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:55.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:55.103 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:48:55.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:48:55.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:55.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:48:55.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:56.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:56.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:56.557 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:48:56.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:48:56.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:56.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:48:57.082 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:58.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:58.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:58.084 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:48:58.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:48:58.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:58.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:48:58.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:48:59.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:48:59.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:48:59.572 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:48:59.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:48:59.572 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:48:59.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:49:00.055 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:01.056 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:01.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:01.057 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:49:01.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:49:01.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:01.057 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:49:01.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:02.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:02.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:02.761 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:49:02.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:49:02.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:02.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:49:03.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:04.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:04.514 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:04.515 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:49:04.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:49:04.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:04.516 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:49:05.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:06.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:06.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:06.385 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:49:06.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:49:06.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:06.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:49:07.139 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:08.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:08.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:08.141 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:49:08.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:49:08.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:08.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:49:08.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:09.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:09.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:09.742 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:49:09.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:49:09.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:09.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:49:10.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:11.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:11.570 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:11.571 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:49:11.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:49:11.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:11.571 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:49:12.093 INFO:tasks.workunit.client.0.smithi080.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3775630618,v1:127.0.0.1:6811/3775630618] [v2:127.0.0.1:6812/3775630618,v1:127.0.0.1:6813/3775630618] exists,up 2ef313ef-752e-46e4-89f8-c3fd253ed8ff 2024-03-25T00:49:12.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:49:12.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:49:12.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:49:12.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:49:12.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-03-25T00:49:12.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:49:12.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:49:12.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T00:49:12.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:49:12.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:49:12.098 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:49:12.099 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:49:12.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:49:12.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:49:12.100 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:49:12.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc4cc00d-8fac-4c45-be47-4d8ba5b91f94 2024-03-25T00:49:12.100 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cc4cc00d-8fac-4c45-be47-4d8ba5b91f94' 2024-03-25T00:49:12.101 INFO:tasks.workunit.client.0.smithi080.stdout:add osd2 cc4cc00d-8fac-4c45-be47-4d8ba5b91f94 2024-03-25T00:49:12.101 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:49:12.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIygBmio8fBxAAQcbKf4ljBeawU/euBHNjbg== 2024-03-25T00:49:12.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIygBmio8fBxAAQcbKf4ljBeawU/euBHNjbg=="}' 2024-03-25T00:49:12.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc4cc00d-8fac-4c45-be47-4d8ba5b91f94 -i td/test-erasure-eio/2/new.json 2024-03-25T00:49:12.824 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:49:12.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-03-25T00:49:12.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIygBmio8fBxAAQcbKf4ljBeawU/euBHNjbg== --osd-uuid cc4cc00d-8fac-4c45-be47-4d8ba5b91f94 2024-03-25T00:49:12.880 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:12.873+0000 7f60ac1c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:12.903 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:12.897+0000 7f60ac1c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:12.905 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:12.901+0000 7f60ac1c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:12.905 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:12.901+0000 7f60ac1c47c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:49:12.905 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:12.901+0000 7f60ac1c47c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:49:12.905 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:12.901+0000 7f60ac1c47c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:49:13.004 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:12.997+0000 7f60ac1c47c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-03-25T00:49:19.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-03-25T00:49:19.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:49:19.407 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd2 key to auth repository 2024-03-25T00:49:19.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T00:49:19.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:49:19.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T00:49:19.975 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:49:19.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:49:19.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:49:19.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:49:19.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:49:20.006 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:20.001+0000 7ff8353e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:20.008 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:20.005+0000 7ff8353e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:20.009 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:20.005+0000 7ff8353e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:20.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T00:49:20.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:49:20.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:49:20.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:49:20.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:49:20.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:20.435 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:49:20.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:49:20.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:20.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:20.577 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:20.573+0000 7ff8353e37c0 -1 Falling back to public interface 2024-03-25T00:49:20.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:21.833 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:21.833 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:21.833 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:49:21.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:49:21.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:21.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:22.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:23.094 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:23.089+0000 7ff8353e37c0 -1 osd.2 0 log_to_monitors true 2024-03-25T00:49:23.285 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:23.285 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:23.285 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:49:23.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:49:23.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:23.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:23.807 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:24.809 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:24.809 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:24.810 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:49:24.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:49:24.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:24.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:25.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:26.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:26.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:26.329 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:49:26.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:49:26.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:26.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:26.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:27.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:27.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:27.847 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:49:27.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:49:27.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:27.847 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:28.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:29.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:29.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:29.434 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:49:29.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:49:29.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:29.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:29.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:30.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:30.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:30.991 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:49:30.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:49:30.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:30.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:31.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:32.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:32.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:32.411 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:49:32.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:49:32.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:32.412 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:32.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:33.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:33.991 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:33.991 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:49:33.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:49:33.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:33.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:34.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:35.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:35.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:35.457 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:49:35.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:49:35.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:35.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:35.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:36.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:36.962 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:36.962 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:49:36.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:49:36.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:36.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:37.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:38.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:38.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:38.759 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:49:38.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:49:38.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:38.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:39.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:40.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:40.590 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:40.590 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:49:40.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:49:40.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:40.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:41.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:42.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:42.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:42.523 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:49:42.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:49:42.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:42.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:43.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:44.334 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:44.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:44.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:49:44.335 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:49:44.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:44.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:44.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:45.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:45.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:45.818 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:49:45.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:49:45.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:45.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:46.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:47.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:47.907 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:47.907 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:49:47.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:49:47.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:47.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:49:48.561 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3726912725,v1:127.0.0.1:6819/3726912725] [v2:127.0.0.1:6820/3726912725,v1:127.0.0.1:6821/3726912725] exists,up cc4cc00d-8fac-4c45-be47-4d8ba5b91f94 2024-03-25T00:49:48.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:49:48.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:49:48.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:49:48.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:49:48.564 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:49:48.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:49:48.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:49:48.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:49:48.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:49:48.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:49:48.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:49:48.565 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:49:48.565 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:49:48.565 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:49:48.566 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:49:48.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:49:48.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:49:48.568 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:49:48.568 INFO:tasks.workunit.client.0.smithi080.stdout:add osd3 defbe9ed-edce-4556-aee6-4eb82eaf18a1 2024-03-25T00:49:48.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=defbe9ed-edce-4556-aee6-4eb82eaf18a1 2024-03-25T00:49:48.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 defbe9ed-edce-4556-aee6-4eb82eaf18a1' 2024-03-25T00:49:48.569 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:49:48.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsygBmwu1UIxAA21EDsMtO6nsWnzqlLnOJsg== 2024-03-25T00:49:48.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsygBmwu1UIxAA21EDsMtO6nsWnzqlLnOJsg=="}' 2024-03-25T00:49:48.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new defbe9ed-edce-4556-aee6-4eb82eaf18a1 -i td/test-erasure-eio/3/new.json 2024-03-25T00:49:49.365 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:49:49.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-03-25T00:49:49.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsygBmwu1UIxAA21EDsMtO6nsWnzqlLnOJsg== --osd-uuid defbe9ed-edce-4556-aee6-4eb82eaf18a1 2024-03-25T00:49:49.415 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:49.412+0000 7f9a1f2637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:49.463 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:49.460+0000 7f9a1f2637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:49.464 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:49.464+0000 7f9a1f2637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:49.465 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:49.464+0000 7f9a1f2637c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:49:49.465 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:49.464+0000 7f9a1f2637c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:49:49.465 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:49.464+0000 7f9a1f2637c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:49:49.537 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:49.536+0000 7f9a1f2637c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-03-25T00:49:56.234 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-03-25T00:49:56.234 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:49:56.235 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd3 key to auth repository 2024-03-25T00:49:56.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T00:49:56.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:49:56.783 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.3 2024-03-25T00:49:56.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T00:49:56.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:49:56.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:49:56.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:49:56.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:49:56.814 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:56.812+0000 7f0ff34887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:56.855 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:56.852+0000 7f0ff34887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:56.856 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:56.856+0000 7f0ff34887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:49:57.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T00:49:57.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:49:57.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:49:57.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:49:57.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:49:57.243 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:57.244 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:49:57.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:49:57.244 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:49:57.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:57.425 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:49:57.424+0000 7f0ff34887c0 -1 Falling back to public interface 2024-03-25T00:49:57.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:49:58.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:49:58.718 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:49:58.719 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:49:58.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:49:58.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:49:58.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:49:59.170 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:00.094 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:00.092+0000 7f0ff34887c0 -1 osd.3 0 log_to_monitors true 2024-03-25T00:50:00.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:00.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:00.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:50:00.172 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:50:00.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:00.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:00.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:01.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:01.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:01.792 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:50:01.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:50:01.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:01.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:02.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:03.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:03.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:03.580 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:50:03.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:50:03.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:03.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:04.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:05.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:05.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:05.192 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:50:05.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:50:05.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:05.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:05.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:06.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:06.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:06.667 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:50:06.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:50:06.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:06.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:07.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:08.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:08.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:08.227 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:50:08.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:50:08.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:08.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:08.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:09.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:09.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:09.797 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:50:09.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:50:09.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:09.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:10.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:11.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:11.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:11.391 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:50:11.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:50:11.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:11.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:11.839 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:12.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:12.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:12.841 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:50:12.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:50:12.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:12.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:13.278 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:14.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:14.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:14.281 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:50:14.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:50:14.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:14.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:15.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:16.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:16.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:16.156 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:50:16.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:50:16.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:16.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:16.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:17.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:17.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:17.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:50:17.840 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:50:17.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:17.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:18.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:19.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:19.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:19.647 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:50:19.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:50:19.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:19.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:20.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:21.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:21.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:21.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:50:21.182 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:50:21.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:21.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:21.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:22.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:22.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:22.909 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:50:22.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:50:22.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:22.909 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:23.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:24.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:24.644 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:24.644 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:50:24.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:50:24.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:24.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:25.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:26.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:26.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:26.452 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:50:26.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:50:26.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:26.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:50:27.041 INFO:tasks.workunit.client.0.smithi080.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/90522020,v1:127.0.0.1:6827/90522020] [v2:127.0.0.1:6828/90522020,v1:127.0.0.1:6829/90522020] exists,up defbe9ed-edce-4556-aee6-4eb82eaf18a1 2024-03-25T00:50:27.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:50:27.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:50:27.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:50:27.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-03-25T00:50:27.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:50:27.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:50:27.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:50:27.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:50:27.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:50:27.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:50:27.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:50:27.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:50:27.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:50:27.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:50:27.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:50:27.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-03-25T00:50:27.048 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:50:27.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dfc9e750-3d4c-4661-a0cf-e41cf048b8d3 2024-03-25T00:50:27.049 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 dfc9e750-3d4c-4661-a0cf-e41cf048b8d3' 2024-03-25T00:50:27.049 INFO:tasks.workunit.client.0.smithi080.stdout:add osd4 dfc9e750-3d4c-4661-a0cf-e41cf048b8d3 2024-03-25T00:50:27.049 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:50:27.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTygBm4a7uAxAAdghbqDYCrxDmTb5yljwcbA== 2024-03-25T00:50:27.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTygBm4a7uAxAAdghbqDYCrxDmTb5yljwcbA=="}' 2024-03-25T00:50:27.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dfc9e750-3d4c-4661-a0cf-e41cf048b8d3 -i td/test-erasure-eio/4/new.json 2024-03-25T00:50:27.641 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:50:27.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-03-25T00:50:27.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTygBm4a7uAxAAdghbqDYCrxDmTb5yljwcbA== --osd-uuid dfc9e750-3d4c-4661-a0cf-e41cf048b8d3 2024-03-25T00:50:27.687 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:27.683+0000 7fb4ed6517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:50:27.689 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:27.687+0000 7fb4ed6517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:50:27.690 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:27.687+0000 7fb4ed6517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:50:27.691 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:27.687+0000 7fb4ed6517c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:50:27.691 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:27.687+0000 7fb4ed6517c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:50:27.691 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:27.687+0000 7fb4ed6517c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-03-25T00:50:27.846 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:27.843+0000 7fb4ed6517c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-03-25T00:50:34.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-03-25T00:50:34.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:50:34.462 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd4 key to auth repository 2024-03-25T00:50:34.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-25T00:50:34.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:50:35.015 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.4 2024-03-25T00:50:35.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-25T00:50:35.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:50:35.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:50:35.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:50:35.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:50:35.046 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:35.043+0000 7f09608ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:50:35.048 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:35.047+0000 7f09608ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:50:35.049 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:35.047+0000 7f09608ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:50:35.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-25T00:50:35.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:50:35.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-25T00:50:35.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:50:35.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:50:35.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:35.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:50:35.426 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:50:35.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:35.426 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:35.633 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:35.631+0000 7f09608ec7c0 -1 Falling back to public interface 2024-03-25T00:50:35.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:36.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:36.840 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:36.840 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:50:36.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:50:36.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:36.841 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:37.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:38.261 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:50:38.259+0000 7f09608ec7c0 -1 osd.4 0 log_to_monitors true 2024-03-25T00:50:38.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:38.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:38.267 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:50:38.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:50:38.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:38.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:39.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:40.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:40.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:40.016 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:50:40.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:50:40.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:40.016 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:40.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:41.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:41.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:41.867 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:50:41.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:50:41.868 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:41.868 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:42.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:43.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:43.545 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:43.545 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:50:43.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:50:43.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:43.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:44.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:45.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:45.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:45.098 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:50:45.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:50:45.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:45.098 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:45.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:46.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:46.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:46.555 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:50:46.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:50:46.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:46.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:47.082 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:48.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:48.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:48.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:50:48.084 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:50:48.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:48.085 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:48.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:49.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:49.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:49.569 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:50:49.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:50:49.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:49.569 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:50.115 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:51.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:51.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:51.117 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:50:51.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:50:51.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:51.118 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:51.777 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:52.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:52.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:52.779 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:50:52.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:50:52.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:52.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:53.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:54.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:54.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:54.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:50:54.538 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:50:54.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:54.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:55.169 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:56.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:56.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:56.171 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:50:56.172 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:50:56.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:56.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:57.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:58.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:58.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:58.161 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:50:58.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:50:58.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:58.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:50:58.698 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:50:59.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:50:59.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:50:59.700 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:50:59.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:50:59.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:50:59.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:51:00.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:01.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:01.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:01.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:51:01.148 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:51:01.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:01.148 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:51:01.948 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:02.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:02.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:02.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:51:02.951 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:51:02.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:02.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-25T00:51:03.546 INFO:tasks.workunit.client.0.smithi080.stdout:osd.4 up in weight 1 up_from 37 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/4246491173,v1:127.0.0.1:6835/4246491173] [v2:127.0.0.1:6836/4246491173,v1:127.0.0.1:6837/4246491173] exists,up dfc9e750-3d4c-4661-a0cf-e41cf048b8d3 2024-03-25T00:51:03.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:51:03.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:51:03.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:51:03.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:51:03.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-03-25T00:51:03.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:51:03.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:51:03.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:51:03.551 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:51:03.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:51:03.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:51:03.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-03-25T00:51:03.553 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:51:03.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=929a180f-4a9e-48e1-a6f8-1ef2e45f162b 2024-03-25T00:51:03.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 929a180f-4a9e-48e1-a6f8-1ef2e45f162b' 2024-03-25T00:51:03.554 INFO:tasks.workunit.client.0.smithi080.stdout:add osd5 929a180f-4a9e-48e1-a6f8-1ef2e45f162b 2024-03-25T00:51:03.554 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:51:03.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3ygBmmnQoIhAAAqf0syCxJvHgRmEces0GHg== 2024-03-25T00:51:03.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3ygBmmnQoIhAAAqf0syCxJvHgRmEces0GHg=="}' 2024-03-25T00:51:03.573 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 929a180f-4a9e-48e1-a6f8-1ef2e45f162b -i td/test-erasure-eio/5/new.json 2024-03-25T00:51:04.329 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:51:04.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-03-25T00:51:04.348 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3ygBmmnQoIhAAAqf0syCxJvHgRmEces0GHg== --osd-uuid 929a180f-4a9e-48e1-a6f8-1ef2e45f162b 2024-03-25T00:51:04.382 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:04.379+0000 7faf6ac9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:04.385 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:04.383+0000 7faf6ac9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:04.386 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:04.383+0000 7faf6ac9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:04.387 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:04.383+0000 7faf6ac9c7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:51:04.387 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:04.383+0000 7faf6ac9c7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:51:04.387 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:04.383+0000 7faf6ac9c7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-03-25T00:51:04.523 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:04.519+0000 7faf6ac9c7c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-03-25T00:51:11.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-03-25T00:51:11.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:51:11.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-25T00:51:11.161 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd5 key to auth repository 2024-03-25T00:51:11.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:51:11.759 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.5 2024-03-25T00:51:11.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-25T00:51:11.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:51:11.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:51:11.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:51:11.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:51:11.792 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:11.791+0000 7f8164ec87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:11.798 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:11.795+0000 7f8164ec87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:11.799 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:11.795+0000 7f8164ec87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:12.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-25T00:51:12.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:51:12.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-25T00:51:12.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:51:12.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:51:12.175 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:51:12.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:12.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:51:12.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:12.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:12.346 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:12.343+0000 7f8164ec87c0 -1 Falling back to public interface 2024-03-25T00:51:12.668 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:13.669 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:13.669 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:13.670 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:51:13.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:51:13.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:13.670 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:14.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:14.922 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:14.919+0000 7f8164ec87c0 -1 osd.5 0 log_to_monitors true 2024-03-25T00:51:15.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:15.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:15.072 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:51:15.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:51:15.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:15.072 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:15.637 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:16.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:16.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:16.639 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:51:16.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:51:16.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:16.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:17.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:18.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:18.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:18.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:51:18.329 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:51:18.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:18.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:18.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:19.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:19.853 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:19.853 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:51:19.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:51:19.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:19.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:20.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:21.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:21.310 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:21.311 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:51:21.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:51:21.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:21.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:21.963 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:22.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:22.965 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:22.965 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:51:22.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:51:22.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:22.966 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:23.411 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:24.413 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:24.413 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:24.413 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:51:24.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:51:24.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:24.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:24.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:25.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:25.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:25.929 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:51:25.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:51:25.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:25.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:26.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:27.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:27.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:27.436 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:51:27.437 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:51:27.437 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:27.437 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:28.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:29.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:29.011 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:29.011 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:51:29.011 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:51:29.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:29.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:29.628 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:30.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:30.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:30.630 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:51:30.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:51:30.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:30.631 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:31.699 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:32.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:32.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:32.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:51:32.701 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:51:32.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:32.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:33.646 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:34.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:34.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:34.648 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:51:34.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:51:34.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:34.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:35.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:36.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:36.427 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:36.427 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:51:36.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:51:36.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:36.428 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:37.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:38.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:38.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:38.510 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:51:38.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:51:38.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:38.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:39.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:40.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:40.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:40.435 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:51:40.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:51:40.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:40.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-25T00:51:40.929 INFO:tasks.workunit.client.0.smithi080.stdout:osd.5 up in weight 1 up_from 46 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3874868049,v1:127.0.0.1:6843/3874868049] [v2:127.0.0.1:6844/3874868049,v1:127.0.0.1:6845/3874868049] exists,up 929a180f-4a9e-48e1-a6f8-1ef2e45f162b 2024-03-25T00:51:40.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:51:40.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:51:40.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-03-25T00:51:40.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-03-25T00:51:40.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:51:40.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:51:40.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:51:40.934 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:51:40.934 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:51:40.934 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:51:40.934 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:51:40.934 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:51:40.935 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:51:40.935 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:51:40.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:51:40.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:51:40.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:51:40.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:51:40.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:51:40.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:51:40.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:51:40.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:51:40.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:51:40.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:51:40.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:51:40.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:51:40.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-03-25T00:51:40.938 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:51:40.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2833063e-33e4-411e-ad14-1850ba6ee656 2024-03-25T00:51:40.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 2833063e-33e4-411e-ad14-1850ba6ee656' 2024-03-25T00:51:40.938 INFO:tasks.workunit.client.0.smithi080.stdout:add osd6 2833063e-33e4-411e-ad14-1850ba6ee656 2024-03-25T00:51:40.938 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:51:40.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCcygBmK6D2OBAAp6hgkO6F1XQwV9Gkhdyckg== 2024-03-25T00:51:40.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCcygBmK6D2OBAAp6hgkO6F1XQwV9Gkhdyckg=="}' 2024-03-25T00:51:40.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2833063e-33e4-411e-ad14-1850ba6ee656 -i td/test-erasure-eio/6/new.json 2024-03-25T00:51:41.757 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:51:41.772 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-03-25T00:51:41.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCcygBmK6D2OBAAp6hgkO6F1XQwV9Gkhdyckg== --osd-uuid 2833063e-33e4-411e-ad14-1850ba6ee656 2024-03-25T00:51:41.808 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:41.806+0000 7fc2653d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:41.810 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:41.806+0000 7fc2653d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:41.811 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:41.806+0000 7fc2653d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:41.812 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:41.810+0000 7fc2653d67c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:51:41.812 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:41.810+0000 7fc2653d67c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:51:41.812 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:41.810+0000 7fc2653d67c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-03-25T00:51:41.912 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:41.910+0000 7fc2653d67c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-03-25T00:51:48.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-03-25T00:51:48.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:51:48.955 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd6 key to auth repository 2024-03-25T00:51:48.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-25T00:51:48.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:51:49.502 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.6 2024-03-25T00:51:49.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-25T00:51:49.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:51:49.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:51:49.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:51:49.504 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:51:49.534 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:49.530+0000 7ff6f2ea97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:49.557 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:49.554+0000 7ff6f2ea97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:49.558 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:49.554+0000 7ff6f2ea97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:51:49.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-25T00:51:49.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:51:49.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-25T00:51:49.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:51:49.901 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:51:49.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:49.902 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:51:49.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:51:49.902 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:49.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:51:50.109 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:50.106+0000 7ff6f2ea97c0 -1 Falling back to public interface 2024-03-25T00:51:50.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:51.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:51.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:51.327 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:51:51.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:51:51.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:51.328 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:51:51.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:52.633 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:51:52.630+0000 7ff6f2ea97c0 -1 osd.6 0 log_to_monitors true 2024-03-25T00:51:52.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:52.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:52.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:51:52.735 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:51:52.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:52.735 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:51:53.402 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:54.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:54.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:54.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:51:54.405 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:51:54.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:54.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:51:55.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:56.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:56.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:56.159 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:51:56.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:51:56.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:56.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:51:56.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:57.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:57.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:57.877 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:51:57.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:51:57.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:57.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:51:58.358 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:51:59.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:51:59.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:51:59.361 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:51:59.361 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:51:59.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:51:59.362 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:51:59.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:00.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:00.821 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:00.822 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:52:00.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:52:00.822 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:00.823 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:01.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:02.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:02.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:02.239 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:52:02.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:52:02.240 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:02.240 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:02.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:03.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:03.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:03.783 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:52:03.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:52:03.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:03.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:04.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:05.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:05.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:05.340 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:52:05.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:52:05.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:05.341 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:05.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:06.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:06.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:06.782 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:52:06.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:52:06.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:06.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:07.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:08.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:08.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:08.393 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:52:08.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:52:08.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:08.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:08.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:09.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:09.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:09.988 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:52:09.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:52:09.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:09.989 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:11.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:12.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:12.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:12.377 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:52:12.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:52:12.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:12.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:12.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:13.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:13.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:13.872 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:52:13.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:52:13.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:13.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:14.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:15.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:15.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:15.678 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:52:15.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:52:15.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:15.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:16.617 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:17.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:17.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:17.620 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:52:17.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:52:17.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:17.621 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:18.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:19.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:19.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:19.065 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:52:19.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:52:19.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:19.066 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-25T00:52:19.525 INFO:tasks.workunit.client.0.smithi080.stdout:osd.6 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1185629106,v1:127.0.0.1:6851/1185629106] [v2:127.0.0.1:6852/1185629106,v1:127.0.0.1:6853/1185629106] exists,up 2833063e-33e4-411e-ad14-1850ba6ee656 2024-03-25T00:52:19.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:52:19.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:52:19.528 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:52:19.528 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-03-25T00:52:19.528 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:52:19.528 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:52:19.528 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:52:19.528 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:19.528 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:19.529 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.0.asok 2024-03-25T00:52:19.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-03-25T00:52:19.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.0.asok log flush 2024-03-25T00:52:19.645 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:52:19.658 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-03-25T00:52:19.665 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:48:10.078+0000 7fe0d5aa47c0 0 load: jerasure load: lrc 2024-03-25T00:52:19.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2024-03-25T00:52:19.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:396: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2024-03-25T00:52:19.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:52:19.666 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-03-25T00:52:19.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-03-25T00:52:19.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-03-25T00:52:19.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-03-25T00:52:19.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-03-25T00:52:19.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-03-25T00:52:20.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-03-25T00:52:20.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-03-25T00:52:21.189 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' already exists 2024-03-25T00:52:21.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:52:22.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-03-25T00:52:22.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:52:22.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:52:22.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:52:22.209 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:52:22.210 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:52:22.210 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:52:22.210 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:52:22.210 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:52:22.211 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:52:22.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:52:22.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:52:22.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:52:22.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:52:22.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:52:22.336 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:52:22.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:52:22.849 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:52:22.849 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:52:22.849 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:52:22.849 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:52:22.849 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:52:22.849 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:52:22.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:52:22.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:22.850 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:52:23.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803826 2024-03-25T00:52:23.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803826 2024-03-25T00:52:23.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826' 2024-03-25T00:52:23.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:23.040 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:52:23.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607597 2024-03-25T00:52:23.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607597 2024-03-25T00:52:23.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607597' 2024-03-25T00:52:23.192 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:23.192 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:52:23.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411368 2024-03-25T00:52:23.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411368 2024-03-25T00:52:23.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607597 2-77309411368' 2024-03-25T00:52:23.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:23.379 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:52:23.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084318 2024-03-25T00:52:23.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084318 2024-03-25T00:52:23.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607597 2-77309411368 3-120259084318' 2024-03-25T00:52:23.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:23.531 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:52:23.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789973 2024-03-25T00:52:23.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789973 2024-03-25T00:52:23.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607597 2-77309411368 3-120259084318 4-158913789973' 2024-03-25T00:52:23.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:23.685 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:52:23.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495630 2024-03-25T00:52:23.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495630 2024-03-25T00:52:23.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607597 2-77309411368 3-120259084318 4-158913789973 5-197568495630' 2024-03-25T00:52:23.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:23.844 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:52:23.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201288 2024-03-25T00:52:23.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201288 2024-03-25T00:52:23.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-51539607597 2-77309411368 3-120259084318 4-158913789973 5-197568495630 6-236223201288' 2024-03-25T00:52:23.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:23.997 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803826 2024-03-25T00:52:23.997 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:23.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:52:23.998 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803826 2024-03-25T00:52:23.998 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:24.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803826 2024-03-25T00:52:24.000 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803826 2024-03-25T00:52:24.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-03-25T00:52:24.001 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:52:24.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803826 -lt 25769803826 2024-03-25T00:52:24.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:24.452 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607597 2024-03-25T00:52:24.453 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:24.454 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:52:24.454 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607597 2024-03-25T00:52:24.455 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:24.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607597 2024-03-25T00:52:24.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607597' 2024-03-25T00:52:24.457 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 51539607597 2024-03-25T00:52:24.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:52:24.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607597 -lt 51539607597 2024-03-25T00:52:24.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:24.885 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411368 2024-03-25T00:52:24.885 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:24.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:52:24.887 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411368 2024-03-25T00:52:24.887 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:24.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411368 2024-03-25T00:52:24.889 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 77309411368 2024-03-25T00:52:24.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411368' 2024-03-25T00:52:24.889 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:52:25.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411368 -lt 77309411368 2024-03-25T00:52:25.280 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:25.280 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084318 2024-03-25T00:52:25.280 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:25.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:52:25.283 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084318 2024-03-25T00:52:25.283 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:25.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084318 2024-03-25T00:52:25.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084318' 2024-03-25T00:52:25.284 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 120259084318 2024-03-25T00:52:25.284 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:52:25.700 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084318 -lt 120259084318 2024-03-25T00:52:25.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:25.701 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:25.701 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-158913789973 2024-03-25T00:52:25.703 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:52:25.704 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-158913789973 2024-03-25T00:52:25.704 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:25.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789973 2024-03-25T00:52:25.705 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 158913789973 2024-03-25T00:52:25.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 158913789973' 2024-03-25T00:52:25.706 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:52:26.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789974 -lt 158913789973 2024-03-25T00:52:26.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:26.160 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-197568495630 2024-03-25T00:52:26.160 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:26.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:52:26.163 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-197568495630 2024-03-25T00:52:26.163 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:26.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495630 2024-03-25T00:52:26.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 197568495630' 2024-03-25T00:52:26.164 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 197568495630 2024-03-25T00:52:26.164 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:52:26.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495630 -lt 197568495630 2024-03-25T00:52:26.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:26.562 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:26.562 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-236223201288 2024-03-25T00:52:26.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-25T00:52:26.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:26.565 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-236223201288 2024-03-25T00:52:26.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201288 2024-03-25T00:52:26.566 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.6 seq 236223201288 2024-03-25T00:52:26.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 236223201288' 2024-03-25T00:52:26.567 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-25T00:52:26.953 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201288 -lt 236223201288 2024-03-25T00:52:26.954 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:52:26.954 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:52:26.954 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:52:27.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:52:27.496 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:52:27.496 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:52:27.497 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:52:27.497 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:52:27.497 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:52:27.497 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:52:27.497 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:52:27.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:52:27.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:52:27.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:52:27.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:52:28.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:52:28.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:52:28.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:52:28.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-03-25T00:52:28.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:52:28.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:52:28.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-03-25T00:52:28.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:52:28.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:52:28.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:52:28.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:52:28.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:52:28.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:52:28.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:52:28.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:52:28.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-03-25T00:52:28.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:399: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-25T00:52:28.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-25T00:52:28.761 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-03-25T00:52:28.761 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:52:28.761 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-03-25T00:52:28.761 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:52:28.761 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:52:29.183 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:52:29.183 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:52:29.183 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:52:29.183 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:52:29.183 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:52:29.183 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-03-25T00:52:29.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-03-25T00:52:29.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-25T00:52:29.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-03-25T00:52:29.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-03-25T00:52:29.184 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-03-25T00:52:29.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-25T00:52:29.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-03-25T00:52:29.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-25T00:52:29.184 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-03-25T00:52:29.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-25T00:52:29.185 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-25T00:52:29.185 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-25T00:52:29.186 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-03-25T00:52:29.186 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-03-25T00:52:29.186 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-03-25T00:52:29.186 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:52:29.187 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:29.187 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:29.187 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.3.asok 2024-03-25T00:52:29.187 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81310/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-03-25T00:52:29.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-25T00:52:29.295 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-25T00:52:29.296 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-25T00:52:29.296 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-03-25T00:52:29.296 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-03-25T00:52:29.296 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-03-25T00:52:29.297 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:52:29.297 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:29.297 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:29.297 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.3.asok 2024-03-25T00:52:29.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-25T00:52:29.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-03-25T00:52:29.422 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2024-03-25T00:52:29.422 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:52:29.422 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-03-25T00:52:29.422 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-03-25T00:52:29.423 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:52:29.843 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:52:29.843 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:52:29.843 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:52:29.843 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:52:29.843 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:52:29.843 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-03-25T00:52:29.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: initial_osds=('3' '5' '1' '0' '6') 2024-03-25T00:52:29.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: local -a initial_osds 2024-03-25T00:52:29.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:402: TEST_ec_single_recovery_error: local last_osd=6 2024-03-25T00:52:29.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:404: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2024-03-25T00:52:29.844 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:52:29.844 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:52:29.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:52:29.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:52:29.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:52:30.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:52:30.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_single_recovery_error: ceph osd down 6 2024-03-25T00:52:30.745 INFO:tasks.workunit.client.0.smithi080.stderr:osd.6 is already down. 2024-03-25T00:52:30.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:406: TEST_ec_single_recovery_error: ceph osd out 6 2024-03-25T00:52:31.452 INFO:tasks.workunit.client.0.smithi080.stderr:osd.6 is already out. 2024-03-25T00:52:31.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:409: TEST_ec_single_recovery_error: wait_for_clean 2024-03-25T00:52:31.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:52:31.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:52:31.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:52:31.470 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:52:31.470 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:52:31.471 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:52:31.471 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:52:31.471 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:52:31.471 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:52:31.574 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:52:31.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:52:31.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:52:31.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:52:31.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:52:31.575 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:52:32.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:52:32.167 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:52:32.167 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:52:32.167 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:52:32.167 INFO:tasks.workunit.client.0.smithi080.stderr:4 2024-03-25T00:52:32.167 INFO:tasks.workunit.client.0.smithi080.stderr:5 2024-03-25T00:52:32.167 INFO:tasks.workunit.client.0.smithi080.stderr:6' 2024-03-25T00:52:32.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:52:32.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:32.168 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:52:32.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803830 2024-03-25T00:52:32.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803830 2024-03-25T00:52:32.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830' 2024-03-25T00:52:32.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:32.484 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:52:32.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607600 2024-03-25T00:52:32.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607600 2024-03-25T00:52:32.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-51539607600' 2024-03-25T00:52:32.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:32.640 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:52:32.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411372 2024-03-25T00:52:32.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411372 2024-03-25T00:52:32.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-51539607600 2-77309411372' 2024-03-25T00:52:32.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:32.888 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:52:33.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084322 2024-03-25T00:52:33.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084322 2024-03-25T00:52:33.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-51539607600 2-77309411372 3-120259084322' 2024-03-25T00:52:33.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:33.041 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-25T00:52:33.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789977 2024-03-25T00:52:33.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789977 2024-03-25T00:52:33.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-51539607600 2-77309411372 3-120259084322 4-158913789977' 2024-03-25T00:52:33.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:33.189 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-25T00:52:33.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495634 2024-03-25T00:52:33.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495634 2024-03-25T00:52:33.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803830 1-51539607600 2-77309411372 3-120259084322 4-158913789977 5-197568495634' 2024-03-25T00:52:33.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:52:33.347 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-25T00:52:33.456 INFO:tasks.workunit.client.0.smithi080.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-03-25T00:52:33.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-03-25T00:52:33.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-03-25T00:52:33.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-03-25T00:52:33.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:33.458 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803830 2024-03-25T00:52:33.459 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:33.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:52:33.460 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803830 2024-03-25T00:52:33.460 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:33.461 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803830 2024-03-25T00:52:33.461 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803830 2024-03-25T00:52:33.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-03-25T00:52:33.462 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:52:33.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803830 -lt 25769803830 2024-03-25T00:52:33.860 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:33.860 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607600 2024-03-25T00:52:33.861 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:33.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:52:33.864 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607600 2024-03-25T00:52:33.864 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607600 2024-03-25T00:52:33.865 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 51539607600 2024-03-25T00:52:33.865 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607600' 2024-03-25T00:52:33.866 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:52:34.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607601 -lt 51539607600 2024-03-25T00:52:34.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:34.322 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411372 2024-03-25T00:52:34.322 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:34.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:52:34.324 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411372 2024-03-25T00:52:34.324 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:34.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411372 2024-03-25T00:52:34.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411372' 2024-03-25T00:52:34.326 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 77309411372 2024-03-25T00:52:34.326 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:52:34.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411372 -lt 77309411372 2024-03-25T00:52:34.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:34.806 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084322 2024-03-25T00:52:34.806 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:34.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:52:34.809 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084322 2024-03-25T00:52:34.809 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:34.810 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084322 2024-03-25T00:52:34.810 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 120259084322 2024-03-25T00:52:34.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084322' 2024-03-25T00:52:34.811 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:52:35.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084322 -lt 120259084322 2024-03-25T00:52:35.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:35.221 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-158913789977 2024-03-25T00:52:35.221 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:35.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-25T00:52:35.223 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-158913789977 2024-03-25T00:52:35.224 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:35.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789977 2024-03-25T00:52:35.225 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.4 seq 158913789977 2024-03-25T00:52:35.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 158913789977' 2024-03-25T00:52:35.226 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-25T00:52:35.641 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789977 -lt 158913789977 2024-03-25T00:52:35.641 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:52:35.641 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-197568495634 2024-03-25T00:52:35.641 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:52:35.642 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-25T00:52:35.643 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-197568495634 2024-03-25T00:52:35.643 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:52:35.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495634 2024-03-25T00:52:35.645 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 197568495634' 2024-03-25T00:52:35.645 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.5 seq 197568495634 2024-03-25T00:52:35.645 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-25T00:52:36.052 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495634 -lt 197568495634 2024-03-25T00:52:36.052 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:52:36.053 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:52:36.053 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:52:36.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:52:36.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:52:36.602 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:52:36.602 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:52:36.602 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:52:36.602 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:52:36.602 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:52:36.603 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:52:37.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T00:52:37.021 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:52:37.022 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:52:37.022 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:52:37.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T00:52:37.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T00:52:37.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T00:52:37.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T00:52:37.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T00:52:37.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T00:52:37.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T00:52:37.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:52:37.658 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:52:37.658 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:52:37.658 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:52:37.658 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:52:37.658 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:52:37.658 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:52:38.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:52:38.051 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:52:38.051 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:52:38.051 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:52:38.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:52:38.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:52:38.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:52:38.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-03-25T00:52:38.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:52:38.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:52:38.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-03-25T00:52:38.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:52:38.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:52:38.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-03-25T00:52:38.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:52:38.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:52:38.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2024-03-25T00:52:38.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:52:38.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-03-25T00:52:39.098 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' does not exist 2024-03-25T00:52:39.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-03-25T00:52:40.004 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile myprofile does not exist 2024-03-25T00:52:40.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-03-25T00:52:40.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:52:40.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:52:40.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:52:40.021 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:52:40.021 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:52:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:52:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:52:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:52:40.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:52:40.337 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:52:40.338 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:52:40.339 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:52:40.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:52:40.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:52:40.341 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:52:40.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:52:40.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:52:40.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:52:40.343 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:52:40.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:52:40.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:52:40.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:52:40.384 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:52:40.384 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:40.384 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:40.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:52:40.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:52:40.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:52:40.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:52:40.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:52:40.390 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:52:40.391 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:52:40.391 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:52:40.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:52:40.392 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:52:40.392 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:52:40.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:52:40.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:52:40.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:52:40.394 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:52:40.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:52:40.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:52:40.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:52:40.398 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:52:40.398 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:40.398 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:40.398 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:52:40.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:52:40.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:52:40.399 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-03-25T00:52:40.400 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:52:40.400 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:40.400 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:40.400 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81310 2024-03-25T00:52:40.404 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:52:40.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:52:40.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:52:40.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-03-25T00:52:40.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-03-25T00:52:40.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-03-25T00:52:40.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:52:40.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:52:40.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:52:40.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-03-25T00:52:40.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-03-25T00:52:40.865 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:52:40.865 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:52:40.866 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:52:40.866 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:52:40.866 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:40.866 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:40.867 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:52:40.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T00:52:41.110 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:52:41.111 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:52:41.111 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:52:41.111 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:52:41.111 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:52:41.112 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:52:41.112 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:52:41.112 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:52:41.112 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:52:41.113 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:52:41.113 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:41.113 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:41.113 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:52:41.113 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:52:41.113 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get fsid 2024-03-25T00:52:41.233 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:52:41.233 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:52:41.234 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:52:41.234 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:52:41.234 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:52:41.235 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:52:41.235 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:52:41.235 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:52:41.235 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:52:41.235 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:41.236 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:41.236 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:52:41.236 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:52:41.236 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get mon_host 2024-03-25T00:52:41.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-03-25T00:52:41.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-03-25T00:52:41.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:52:41.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:52:41.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:52:41.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-03-25T00:52:41.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:52:42.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:52:42.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:52:42.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:52:42.292 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:52:42.292 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:42.292 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:42.292 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:52:42.293 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:52:42.294 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:52:42.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-03-25T00:52:42.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T00:52:42.655 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' already exists 2024-03-25T00:52:42.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:52:43.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-03-25T00:52:43.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:52:43.675 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:52:43.675 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:52:43.675 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:43.676 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:43.676 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:52:43.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-03-25T00:52:43.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-mon.a.asok log flush 2024-03-25T00:52:43.773 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:52:43.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-03-25T00:52:43.788 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:52:40.905+0000 7faa45ef1c80 0 load: jerasure load: lrc 2024-03-25T00:52:43.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2024-03-25T00:52:43.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2024-03-25T00:52:43.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_bad_size_shard_0: setup_osds 4 2024-03-25T00:52:43.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-03-25T00:52:43.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-03-25T00:52:43.789 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-03-25T00:52:43.790 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-03-25T00:52:43.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:52:43.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-03-25T00:52:43.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:52:43.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:52:43.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:52:43.795 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:52:43.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:52:43.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:52:43.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:52:43.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:52:43.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-03-25T00:52:43.797 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:52:43.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8670d4b2-8715-40b6-ad6d-2a1d9d884a55 2024-03-25T00:52:43.798 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 8670d4b2-8715-40b6-ad6d-2a1d9d884a55 2024-03-25T00:52:43.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8670d4b2-8715-40b6-ad6d-2a1d9d884a55' 2024-03-25T00:52:43.798 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:52:43.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDbygBmwCp2MBAA7asP5MjQFLH8kP2rlGxhXg== 2024-03-25T00:52:43.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDbygBmwCp2MBAA7asP5MjQFLH8kP2rlGxhXg=="}' 2024-03-25T00:52:43.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8670d4b2-8715-40b6-ad6d-2a1d9d884a55 -i td/test-erasure-eio/0/new.json 2024-03-25T00:52:44.156 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:52:44.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-03-25T00:52:44.174 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbygBmwCp2MBAA7asP5MjQFLH8kP2rlGxhXg== --osd-uuid 8670d4b2-8715-40b6-ad6d-2a1d9d884a55 2024-03-25T00:52:44.205 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:44.201+0000 7f7b22bc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:52:44.242 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:44.237+0000 7f7b22bc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:52:44.244 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:44.241+0000 7f7b22bc97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:52:44.244 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:44.241+0000 7f7b22bc97c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:52:44.244 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:44.241+0000 7f7b22bc97c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:52:44.245 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:44.241+0000 7f7b22bc97c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:52:44.281 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:44.277+0000 7f7b22bc97c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-03-25T00:52:52.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-03-25T00:52:52.375 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:52:52.376 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:52:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:52:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:52:52.950 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:52:52.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:52:52.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:52:52.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:52:52.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:52:52.951 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:52:52.982 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:52.977+0000 7f90173ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:52:52.984 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:52.981+0000 7f90173ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:52:52.986 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:52.981+0000 7f90173ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:52:53.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:52:53.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:52:53.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:52:53.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:52:53.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:52:53.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:53.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:52:53.333 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:52:53.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:53.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:52:53.530 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:53.525+0000 7f90173ac7c0 -1 Falling back to public interface 2024-03-25T00:52:53.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:54.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:54.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:54.762 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:52:54.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:52:54.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:54.763 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:52:55.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:56.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:56.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:56.160 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:52:56.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:52:56.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:56.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:52:56.240 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:52:56.237+0000 7f90173ac7c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:52:56.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:57.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:57.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:57.599 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:52:57.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:52:57.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:57.599 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:52:58.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:52:59.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:52:59.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:52:59.132 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:52:59.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:52:59.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:52:59.133 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:52:59.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:00.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:00.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:00.595 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:53:00.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:53:00.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:00.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:01.015 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:02.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:02.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:02.017 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:53:02.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:53:02.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:02.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:02.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:03.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:03.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:03.652 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:53:03.652 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:53:03.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:03.653 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:04.256 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:05.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:05.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:05.258 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:53:05.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:53:05.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:05.259 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:05.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:06.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:06.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:06.753 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:53:06.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:53:06.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:06.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:07.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:08.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:08.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:08.271 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:53:08.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:53:08.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:08.272 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:08.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:09.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:09.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:09.696 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:53:09.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:53:09.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:09.697 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:10.591 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:11.592 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:11.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:11.593 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:53:11.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:53:11.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:11.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:12.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:13.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:13.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:13.211 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:53:13.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:53:13.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:13.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:14.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:15.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:15.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:15.020 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:53:15.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:53:15.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:15.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:16.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:17.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:17.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:17.142 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:53:17.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:53:17.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:17.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:17.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:18.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:18.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:18.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:53:18.877 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:53:18.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:18.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:20.075 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:21.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:21.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:21.077 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:53:21.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:53:21.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:21.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:22.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:23.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:23.502 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:23.503 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:53:23.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:53:23.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:23.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:24.278 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:25.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:25.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:25.283 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-25T00:53:25.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T00:53:25.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:25.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:26.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:27.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:27.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:27.205 INFO:tasks.workunit.client.0.smithi080.stdout:20 2024-03-25T00:53:27.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T00:53:27.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:27.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:53:27.824 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/977652319,v1:127.0.0.1:6803/977652319] [v2:127.0.0.1:6804/977652319,v1:127.0.0.1:6805/977652319] exists,up 8670d4b2-8715-40b6-ad6d-2a1d9d884a55 2024-03-25T00:53:27.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:53:27.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:53:27.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:53:27.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:53:27.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-03-25T00:53:27.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:53:27.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:53:27.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T00:53:27.826 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:53:27.827 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:53:27.828 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:53:27.828 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:53:27.828 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:53:27.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:53:27.830 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:53:27.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8e4d9e43-8bdb-4aca-b79c-aeeb9131c594 2024-03-25T00:53:27.831 INFO:tasks.workunit.client.0.smithi080.stdout:add osd1 8e4d9e43-8bdb-4aca-b79c-aeeb9131c594 2024-03-25T00:53:27.831 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8e4d9e43-8bdb-4aca-b79c-aeeb9131c594' 2024-03-25T00:53:27.832 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:53:27.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHywBmmoPnMhAAyJAzjXrmcAV6xdkD2J2gug== 2024-03-25T00:53:27.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHywBmmoPnMhAAyJAzjXrmcAV6xdkD2J2gug=="}' 2024-03-25T00:53:27.854 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8e4d9e43-8bdb-4aca-b79c-aeeb9131c594 -i td/test-erasure-eio/1/new.json 2024-03-25T00:53:28.826 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:53:28.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-03-25T00:53:28.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHywBmmoPnMhAAyJAzjXrmcAV6xdkD2J2gug== --osd-uuid 8e4d9e43-8bdb-4aca-b79c-aeeb9131c594 2024-03-25T00:53:28.888 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:28.888+0000 7f6a744907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:53:28.916 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:28.916+0000 7f6a744907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:53:28.918 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:28.916+0000 7f6a744907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:53:28.918 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:28.916+0000 7f6a744907c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:53:28.918 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:28.916+0000 7f6a744907c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:53:28.918 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:28.916+0000 7f6a744907c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:53:28.995 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:28.992+0000 7f6a744907c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-03-25T00:53:36.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-03-25T00:53:36.805 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:53:36.805 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd1 key to auth repository 2024-03-25T00:53:36.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T00:53:36.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:53:37.372 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.1 2024-03-25T00:53:37.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T00:53:37.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:53:37.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:53:37.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:53:37.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:53:37.404 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:37.404+0000 7f3e7764d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:53:37.441 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:37.440+0000 7f3e7764d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:53:37.442 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:37.440+0000 7f3e7764d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:53:37.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T00:53:37.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:53:37.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:53:37.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:53:37.819 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:53:37.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:37.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:53:37.820 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:53:37.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:37.820 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:38.016 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:38.016+0000 7f3e7764d7c0 -1 Falling back to public interface 2024-03-25T00:53:38.274 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:39.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:39.276 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:39.276 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:53:39.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:53:39.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:39.277 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:39.738 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:40.739 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:40.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:40.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:53:40.740 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:53:40.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:40.740 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:40.848 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:53:40.848+0000 7f3e7764d7c0 -1 osd.1 0 log_to_monitors true 2024-03-25T00:53:41.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:42.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:42.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:42.302 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:53:42.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:53:42.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:42.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:42.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:43.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:43.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:43.766 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:53:43.767 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:53:43.767 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:43.767 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:44.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:45.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:45.457 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:45.457 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:53:45.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:53:45.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:45.458 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:45.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:47.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:47.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:47.002 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:53:47.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:53:47.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:47.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:47.480 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:48.481 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:48.482 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:48.482 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:53:48.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:53:48.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:48.483 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:49.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:50.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:50.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:50.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:53:50.084 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:53:50.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:50.084 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:50.630 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:51.632 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:51.632 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:51.632 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:53:51.633 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:53:51.633 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:51.633 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:52.224 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:53.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:53.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:53.226 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:53:53.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:53:53.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:53.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:53.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:54.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:54.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:54.754 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:53:54.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:53:54.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:54.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:55.344 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:56.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:56.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:56.346 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:53:56.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:53:56.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:56.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:57.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:58.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:58.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:58.012 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:53:58.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:53:58.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:58.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:53:58.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:53:59.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:53:59.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:53:59.492 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:53:59.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:53:59.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:53:59.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:54:00.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:01.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:01.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:01.525 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:54:01.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:54:01.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:01.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:54:02.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:03.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:03.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:03.957 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:54:03.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:54:03.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:03.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:54:04.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:05.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:05.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:05.949 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:54:05.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:54:05.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:05.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:54:06.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:07.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:07.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:07.781 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:54:07.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:54:07.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:07.782 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:54:08.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:09.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:09.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:09.553 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-25T00:54:09.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T00:54:09.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:09.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:54:10.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:11.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:11.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:11.728 INFO:tasks.workunit.client.0.smithi080.stdout:20 2024-03-25T00:54:11.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T00:54:11.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:11.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:54:12.299 INFO:tasks.workunit.client.0.smithi080.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2115114859,v1:127.0.0.1:6811/2115114859] [v2:127.0.0.1:6812/2115114859,v1:127.0.0.1:6813/2115114859] exists,up 8e4d9e43-8bdb-4aca-b79c-aeeb9131c594 2024-03-25T00:54:12.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:54:12.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:54:12.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:54:12.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:54:12.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-03-25T00:54:12.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:54:12.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:54:12.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T00:54:12.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:54:12.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:54:12.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:54:12.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:54:12.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:54:12.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:54:12.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:54:12.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:54:12.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:54:12.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:54:12.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:54:12.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:54:12.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:54:12.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:54:12.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:54:12.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:54:12.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:54:12.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:54:12.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:54:12.306 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:54:12.307 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:54:12.307 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d6e0680e-6b7d-4460-8bed-c15238384027 2024-03-25T00:54:12.307 INFO:tasks.workunit.client.0.smithi080.stdout:add osd2 d6e0680e-6b7d-4460-8bed-c15238384027 2024-03-25T00:54:12.308 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d6e0680e-6b7d-4460-8bed-c15238384027' 2024-03-25T00:54:12.308 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:54:12.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA0ywBm9a6yExAAVu6w6J60UJ8Vz4hFtIhSWg== 2024-03-25T00:54:12.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA0ywBm9a6yExAAVu6w6J60UJ8Vz4hFtIhSWg=="}' 2024-03-25T00:54:12.329 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d6e0680e-6b7d-4460-8bed-c15238384027 -i td/test-erasure-eio/2/new.json 2024-03-25T00:54:13.757 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:54:13.778 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-03-25T00:54:13.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0ywBm9a6yExAAVu6w6J60UJ8Vz4hFtIhSWg== --osd-uuid d6e0680e-6b7d-4460-8bed-c15238384027 2024-03-25T00:54:13.818 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:13.819+0000 7f97346a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:54:13.896 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:13.895+0000 7f97346a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:54:13.898 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:13.899+0000 7f97346a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:54:13.899 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:13.899+0000 7f97346a97c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:54:13.899 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:13.899+0000 7f97346a97c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:54:13.899 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:13.899+0000 7f97346a97c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:54:14.054 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:14.055+0000 7f97346a97c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-03-25T00:54:21.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-03-25T00:54:21.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:54:21.337 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd2 key to auth repository 2024-03-25T00:54:21.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T00:54:21.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:54:21.915 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T00:54:21.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T00:54:21.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:54:21.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:54:21.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:54:21.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:54:21.941 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:21.939+0000 7f0a404517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:54:21.943 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:21.943+0000 7f0a404517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:54:21.944 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:21.943+0000 7f0a404517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:54:22.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T00:54:22.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:54:22.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T00:54:22.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:54:22.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:54:22.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:22.314 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:54:22.315 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:54:22.315 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:22.315 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:22.491 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:22.491+0000 7f0a404517c0 -1 Falling back to public interface 2024-03-25T00:54:22.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:23.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:23.750 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:23.751 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:54:23.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:54:23.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:23.751 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:24.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:25.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:25.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:25.159 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:54:25.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:54:25.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:25.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:25.237 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:25.239+0000 7f0a404517c0 -1 osd.2 0 log_to_monitors true 2024-03-25T00:54:25.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:26.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:26.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:26.711 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:54:26.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:54:26.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:26.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:27.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:28.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:28.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:28.230 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:54:28.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:54:28.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:28.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:28.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:29.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:29.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:29.743 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:54:29.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:54:29.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:29.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:30.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:31.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:31.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:31.332 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:54:31.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:54:31.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:31.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:31.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:32.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:32.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:32.889 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:54:32.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:54:32.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:32.889 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:33.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:34.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:34.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:34.552 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:54:34.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:54:34.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:34.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:34.980 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:35.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:35.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:35.982 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:54:35.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:54:35.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:35.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:36.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:37.533 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:37.533 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:37.534 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:54:37.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:54:37.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:37.534 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:38.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:39.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:39.026 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:39.026 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:54:39.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:54:39.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:39.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:39.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:40.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:40.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:40.683 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:54:40.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:54:40.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:40.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:41.515 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:42.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:42.517 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:42.517 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:54:42.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:54:42.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:42.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:43.337 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:44.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:44.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:44.342 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:54:44.343 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:54:44.343 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:44.343 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:45.288 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:46.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:46.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:46.290 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:54:46.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:54:46.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:46.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:47.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:48.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:48.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:48.520 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:54:48.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:54:48.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:48.521 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:49.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:50.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:50.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:50.549 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:54:50.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:54:50.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:50.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:51.393 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:52.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:52.395 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:52.395 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:54:52.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:54:52.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:52.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:53.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:54:54.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:54:54.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:54:54.406 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-25T00:54:54.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T00:54:54.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:54:54.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T00:54:55.242 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2698821488,v1:127.0.0.1:6819/2698821488] [v2:127.0.0.1:6820/2698821488,v1:127.0.0.1:6821/2698821488] exists,up d6e0680e-6b7d-4460-8bed-c15238384027 2024-03-25T00:54:55.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:54:55.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:54:55.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:54:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:54:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-03-25T00:54:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:54:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:54:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T00:54:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:54:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:54:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:54:55.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:54:55.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:54:55.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:54:55.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:54:55.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:54:55.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:54:55.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:54:55.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:54:55.248 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:54:55.248 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:54:55.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:54:55.249 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:54:55.249 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:54:55.249 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:54:55.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:54:55.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:54:55.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:54:55.251 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:54:55.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7747434-5bbc-4eef-8999-bff773a9396a 2024-03-25T00:54:55.252 INFO:tasks.workunit.client.0.smithi080.stdout:add osd3 d7747434-5bbc-4eef-8999-bff773a9396a 2024-03-25T00:54:55.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 d7747434-5bbc-4eef-8999-bff773a9396a' 2024-03-25T00:54:55.252 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:54:55.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfywBm8jMcEBAAd5iyN2KZkVrd/tfnE5mYpw== 2024-03-25T00:54:55.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfywBm8jMcEBAAd5iyN2KZkVrd/tfnE5mYpw=="}' 2024-03-25T00:54:55.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7747434-5bbc-4eef-8999-bff773a9396a -i td/test-erasure-eio/3/new.json 2024-03-25T00:54:56.756 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:54:56.774 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-03-25T00:54:56.775 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfywBm8jMcEBAAd5iyN2KZkVrd/tfnE5mYpw== --osd-uuid d7747434-5bbc-4eef-8999-bff773a9396a 2024-03-25T00:54:56.808 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:56.806+0000 7fe82f3327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:54:56.845 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:56.846+0000 7fe82f3327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:54:56.846 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:56.846+0000 7fe82f3327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:54:56.847 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:56.846+0000 7fe82f3327c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:54:56.847 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:56.846+0000 7fe82f3327c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:54:56.847 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:56.846+0000 7fe82f3327c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T00:54:56.953 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:54:56.954+0000 7fe82f3327c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-03-25T00:55:04.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-03-25T00:55:04.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:55:04.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T00:55:04.755 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd3 key to auth repository 2024-03-25T00:55:04.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:55:05.334 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.3 2024-03-25T00:55:05.335 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T00:55:05.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:55:05.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:55:05.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:55:05.336 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:55:05.367 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:55:05.366+0000 7fdd4f0c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:55:05.369 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:55:05.370+0000 7fdd4f0c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:55:05.370 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:55:05.370+0000 7fdd4f0c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:55:05.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T00:55:05.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:55:05.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:55:05.747 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:55:05.747 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:55:05.747 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:05.747 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:55:05.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:55:05.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:05.748 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:05.930 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:55:05.930+0000 7fdd4f0c87c0 -1 Falling back to public interface 2024-03-25T00:55:06.145 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:07.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:07.147 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:07.148 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:55:07.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:55:07.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:07.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:07.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:08.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:08.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:08.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:55:08.560 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:55:08.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:08.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:08.591 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:55:08.590+0000 7fdd4f0c87c0 -1 osd.3 0 log_to_monitors true 2024-03-25T00:55:09.069 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:10.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:10.070 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:10.071 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:55:10.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:55:10.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:10.071 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:10.657 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:11.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:11.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:11.659 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:55:11.659 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:55:11.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:11.660 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:12.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:13.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:13.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:13.387 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:55:13.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:55:13.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:13.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:13.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:14.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:14.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:14.973 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:55:14.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:55:14.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:14.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:15.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:16.417 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:16.418 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:16.418 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:55:16.418 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:55:16.418 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:16.418 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:17.117 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:18.119 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:18.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:18.120 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:55:18.120 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:55:18.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:18.121 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:18.556 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:19.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:19.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:19.558 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:55:19.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:55:19.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:19.559 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:20.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:21.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:21.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:21.206 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:55:21.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:55:21.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:21.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:21.626 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:22.628 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:22.628 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:22.628 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:55:22.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:55:22.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:22.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:23.281 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:24.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:24.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:24.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:55:24.283 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:55:24.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:24.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:24.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:25.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:25.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:25.869 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:55:25.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:55:25.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:25.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:26.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:27.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:27.468 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:27.468 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:55:27.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:55:27.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:27.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:28.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:29.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:29.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:29.144 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:55:29.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:55:29.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:29.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:30.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:31.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:31.050 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:31.050 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:55:31.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:55:31.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:31.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:31.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:32.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:32.694 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:32.694 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:55:32.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:55:32.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:32.695 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:33.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:34.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:34.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:34.595 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:55:34.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:55:34.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:34.596 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:35.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:36.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:36.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:36.449 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-25T00:55:36.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T00:55:36.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:36.450 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:37.260 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:38.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:38.262 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:38.262 INFO:tasks.workunit.client.0.smithi080.stdout:20 2024-03-25T00:55:38.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T00:55:38.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:38.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:39.019 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:40.020 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:40.021 INFO:tasks.workunit.client.0.smithi080.stdout:21 2024-03-25T00:55:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-25T00:55:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:40.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:40.928 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:41.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:41.931 INFO:tasks.workunit.client.0.smithi080.stdout:22 2024-03-25T00:55:41.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:41.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-25T00:55:41.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:41.932 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:42.382 INFO:tasks.workunit.client.0.smithi080.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1518524382,v1:127.0.0.1:6827/1518524382] [v2:127.0.0.1:6828/1518524382,v1:127.0.0.1:6829/1518524382] exists,up d7747434-5bbc-4eef-8999-bff773a9396a 2024-03-25T00:55:42.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:55:42.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:55:42.384 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:55:42.384 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-03-25T00:55:42.385 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T00:55:42.385 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T00:55:42.385 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:55:42.385 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:55:42.385 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:55:42.385 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.0.asok 2024-03-25T00:55:42.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-03-25T00:55:42.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.0.asok log flush 2024-03-25T00:55:42.480 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:55:42.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-03-25T00:55:42.497 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:52:53.533+0000 7f90173ac7c0 0 load: jerasure load: lrc 2024-03-25T00:55:42.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2024-03-25T00:55:42.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-03-25T00:55:42.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:55:42.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-03-25T00:55:42.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-03-25T00:55:42.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-03-25T00:55:42.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-03-25T00:55:42.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-03-25T00:55:42.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-03-25T00:55:43.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-03-25T00:55:43.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-03-25T00:55:45.012 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' already exists 2024-03-25T00:55:45.032 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:55:46.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-03-25T00:55:46.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:55:46.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:55:46.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:55:46.035 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:55:46.036 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:55:46.036 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:55:46.037 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:55:46.037 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:55:46.037 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:55:46.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:55:46.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:55:46.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:55:46.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:55:46.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:55:46.167 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:55:46.636 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:55:46.636 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:55:46.636 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:55:46.636 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T00:55:46.636 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:55:46.636 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:55:46.636 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:55:46.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803810 2024-03-25T00:55:46.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803810 2024-03-25T00:55:46.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810' 2024-03-25T00:55:46.935 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:55:46.935 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:55:47.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574878 2024-03-25T00:55:47.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574878 2024-03-25T00:55:47.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810 1-55834574878' 2024-03-25T00:55:47.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:55:47.090 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:55:47.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378643 2024-03-25T00:55:47.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378643 2024-03-25T00:55:47.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810 1-55834574878 2-81604378643' 2024-03-25T00:55:47.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:55:47.227 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:55:47.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084302 2024-03-25T00:55:47.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084302 2024-03-25T00:55:47.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803810 1-55834574878 2-81604378643 3-120259084302' 2024-03-25T00:55:47.377 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:55:47.378 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803810 2024-03-25T00:55:47.378 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:55:47.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:55:47.379 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803810 2024-03-25T00:55:47.379 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:55:47.380 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803810 2024-03-25T00:55:47.380 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803810 2024-03-25T00:55:47.381 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803810' 2024-03-25T00:55:47.381 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:55:47.833 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803810 -lt 25769803810 2024-03-25T00:55:47.834 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:55:47.834 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574878 2024-03-25T00:55:47.834 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:55:47.836 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:55:47.837 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574878 2024-03-25T00:55:47.837 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:55:47.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574878 2024-03-25T00:55:47.838 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574878' 2024-03-25T00:55:47.838 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 55834574878 2024-03-25T00:55:47.839 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:55:48.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574876 -lt 55834574878 2024-03-25T00:55:48.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:55:49.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:55:49.326 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:55:49.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574879 -lt 55834574878 2024-03-25T00:55:49.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:55:49.742 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378643 2024-03-25T00:55:49.742 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:55:49.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:55:49.744 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378643 2024-03-25T00:55:49.744 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:55:49.745 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378643 2024-03-25T00:55:49.746 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 81604378643 2024-03-25T00:55:49.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378643' 2024-03-25T00:55:49.747 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:55:50.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378643 -lt 81604378643 2024-03-25T00:55:50.203 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:55:50.204 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084302 2024-03-25T00:55:50.204 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:55:50.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:55:50.207 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084302 2024-03-25T00:55:50.207 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:55:50.208 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084302 2024-03-25T00:55:50.208 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 120259084302 2024-03-25T00:55:50.209 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084302' 2024-03-25T00:55:50.209 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:55:50.647 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084302 -lt 120259084302 2024-03-25T00:55:50.647 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:55:50.648 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:55:50.648 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:55:51.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:55:51.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:55:51.230 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:55:51.230 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:55:51.230 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:55:51.230 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:55:51.230 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:55:51.230 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:55:51.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:55:51.610 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:55:51.611 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:55:51.611 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:55:52.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:55:52.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:55:52.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:55:52.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_bad_size_shard_0: local shard_id=0 2024-03-25T00:55:52.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2024-03-25T00:55:52.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-03-25T00:55:52.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-03-25T00:55:52.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-03-25T00:55:52.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-03-25T00:55:52.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-03-25T00:55:52.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-03-25T00:55:52.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-81310-0-10 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81310-0-10 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81310-0-10 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:55:52.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:55:52.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81310-0-10 td/test-erasure-eio/ORIGINAL 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-81310-0-10 td/test-erasure-eio 0 10 set 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81310-0-10 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-03-25T00:55:52.376 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81310-0-10 2024-03-25T00:55:52.377 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:55:52.377 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81310-0-10 2024-03-25T00:55:52.377 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:55:52.377 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81310-0-10 2024-03-25T00:55:52.807 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:55:52.807 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:55:52.807 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-25T00:55:52.807 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-03-25T00:55:52.807 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-03-25T00:55:52.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-03-25T00:55:52.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-03-25T00:55:52.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-03-25T00:55:53.542 INFO:tasks.workunit.client.0.smithi080.stderr:noout is set 2024-03-25T00:55:53.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-03-25T00:55:53.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-03-25T00:55:53.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-03-25T00:55:53.566 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records in 2024-03-25T00:55:53.566 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records out 2024-03-25T00:55:53.567 INFO:tasks.workunit.client.0.smithi080.stderr:10 bytes copied, 0.000186285 s, 53.7 kB/s 2024-03-25T00:55:53.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81310-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:55:53.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-03-25T00:55:53.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:55:53.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T00:55:53.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:55:53.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81310-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:55:53.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-03-25T00:55:53.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:55:53.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T00:55:53.567 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:55:53.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-03-25T00:55:53.568 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:55:53.568 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:55:53.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:55:53.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:55:53.568 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:55:54.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:55:54.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81310-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:55:54.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-03-25T00:55:54.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:55:54.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T00:55:54.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:55:54.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-03-25T00:55:54.879 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81310-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:55:57.210 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:55:57.211 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:55:57.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:55:57.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:55:57.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:55:57.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:55:57.212 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:55:57.212 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:55:57.212 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:55:57.212 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:55:57.213 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:55:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:55:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:55:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:55:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:55:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:55:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:55:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:55:57.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:55:57.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:55:57.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:55:57.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:55:57.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T00:55:57.214 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.3 2024-03-25T00:55:57.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:55:57.215 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-03-25T00:55:57.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T00:55:57.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:55:57.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:55:57.217 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:55:57.249 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:55:57.252+0000 7fa1433ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:55:57.251 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:55:57.256+0000 7fa1433ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:55:57.253 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:55:57.256+0000 7fa1433ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:55:57.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T00:55:57.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:55:57.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:55:57.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:55:57.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:55:57.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:57.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:55:57.685 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:55:57.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:57.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:57.808 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:55:57.812+0000 7fa1433ad7c0 -1 Falling back to public interface 2024-03-25T00:55:58.086 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:55:59.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:55:59.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:55:59.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:55:59.088 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:55:59.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:55:59.089 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:55:59.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:56:00.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:56:00.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:00.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:56:00.560 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:56:00.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:00.560 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:56:00.593 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:00.596+0000 7fa1433ad7c0 -1 osd.3 36 log_to_monitors true 2024-03-25T00:56:01.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:56:02.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:56:02.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:02.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:56:02.167 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:56:02.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:02.167 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:56:02.881 INFO:tasks.workunit.client.0.smithi080.stderr:osd.3 up in weight 1 up_from 42 up_thru 42 down_at 39 last_clean_interval [28,36) [v2:127.0.0.1:6826/40051370,v1:127.0.0.1:6827/40051370] [v2:127.0.0.1:6828/40051370,v1:127.0.0.1:6829/40051370] exists,up d7747434-5bbc-4eef-8999-bff773a9396a 2024-03-25T00:56:02.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:56:02.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:56:02.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:56:02.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:56:02.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:56:02.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:56:02.882 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:56:02.883 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:56:02.883 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:56:02.883 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:56:02.883 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:56:02.883 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:56:02.883 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:56:03.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:56:03.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:56:03.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:56:03.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:56:03.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:56:03.010 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:56:03.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:56:03.430 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:56:03.430 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:56:03.430 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T00:56:03.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:56:03.431 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:03.431 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:56:03.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803816 2024-03-25T00:56:03.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803816 2024-03-25T00:56:03.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803816' 2024-03-25T00:56:03.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:03.689 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:56:03.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574884 2024-03-25T00:56:03.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574884 2024-03-25T00:56:03.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803816 1-55834574884' 2024-03-25T00:56:03.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:03.850 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:56:04.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378648 2024-03-25T00:56:04.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378648 2024-03-25T00:56:04.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803816 1-55834574884 2-81604378648' 2024-03-25T00:56:04.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:04.009 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:56:04.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626435 2024-03-25T00:56:04.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626435 2024-03-25T00:56:04.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803816 1-55834574884 2-81604378648 3-180388626435' 2024-03-25T00:56:04.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:04.163 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803816 2024-03-25T00:56:04.163 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:04.164 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:56:04.165 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803816 2024-03-25T00:56:04.165 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:04.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803816 2024-03-25T00:56:04.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-03-25T00:56:04.166 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 25769803816 2024-03-25T00:56:04.167 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:56:04.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803814 -lt 25769803816 2024-03-25T00:56:04.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:56:05.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:56:05.585 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:56:05.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803816 -lt 25769803816 2024-03-25T00:56:05.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:05.995 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574884 2024-03-25T00:56:05.995 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:05.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:56:05.998 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574884 2024-03-25T00:56:05.998 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:06.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574884 2024-03-25T00:56:06.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574884' 2024-03-25T00:56:06.000 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 55834574884 2024-03-25T00:56:06.000 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:56:06.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574884 -lt 55834574884 2024-03-25T00:56:06.434 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:06.435 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378648 2024-03-25T00:56:06.435 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:06.437 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:56:06.438 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:06.438 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378648 2024-03-25T00:56:06.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378648 2024-03-25T00:56:06.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378648' 2024-03-25T00:56:06.439 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 81604378648 2024-03-25T00:56:06.440 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:56:06.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378648 -lt 81604378648 2024-03-25T00:56:06.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:06.886 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-180388626435 2024-03-25T00:56:06.887 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:06.888 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:56:06.889 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-180388626435 2024-03-25T00:56:06.889 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:06.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626435 2024-03-25T00:56:06.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 180388626435' 2024-03-25T00:56:06.890 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 180388626435 2024-03-25T00:56:06.891 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:56:07.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626436 -lt 180388626435 2024-03-25T00:56:07.322 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:56:07.322 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:56:07.323 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:56:07.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:56:07.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:56:07.870 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:56:07.870 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:56:07.870 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:56:07.870 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:56:07.871 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:56:07.871 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:56:08.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T00:56:08.272 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:56:08.272 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:56:08.272 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:56:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T00:56:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T00:56:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T00:56:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T00:56:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T00:56:08.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T00:56:08.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T00:56:08.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:56:08.906 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:56:08.906 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:56:08.906 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:56:08.906 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:56:08.907 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:56:08.907 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:56:09.333 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:56:09.333 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:56:09.334 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:56:09.334 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:56:09.905 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:56:09.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:56:09.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:56:09.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-03-25T00:56:09.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-03-25T00:56:10.673 INFO:tasks.workunit.client.0.smithi080.stderr:noout is unset 2024-03-25T00:56:10.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81310-0-10 2024-03-25T00:56:10.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:56:10.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:56:10.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81310-0-10 2024-03-25T00:56:10.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:56:10.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:56:10.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81310-0-10 td/test-erasure-eio/COPY 2024-03-25T00:56:11.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:56:11.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:56:11.041 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-03-25T00:56:11.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-03-25T00:56:11.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-81310-0-10 td/test-erasure-eio 1 10 set 2024-03-25T00:56:11.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81310-0-10 2024-03-25T00:56:11.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-03-25T00:56:11.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-03-25T00:56:11.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-03-25T00:56:11.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-03-25T00:56:11.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-03-25T00:56:11.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-03-25T00:56:11.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-03-25T00:56:11.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-03-25T00:56:11.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-03-25T00:56:11.045 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81310-0-10 2024-03-25T00:56:11.045 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:56:11.045 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81310-0-10 2024-03-25T00:56:11.045 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81310-0-10 2024-03-25T00:56:11.045 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:56:11.531 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:56:11.531 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:56:11.531 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-25T00:56:11.531 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-03-25T00:56:11.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-03-25T00:56:11.531 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-03-25T00:56:11.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-03-25T00:56:11.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-03-25T00:56:12.363 INFO:tasks.workunit.client.0.smithi080.stderr:noout is set 2024-03-25T00:56:12.385 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-03-25T00:56:12.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-03-25T00:56:12.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-03-25T00:56:12.387 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records in 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records out 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:10 bytes copied, 0.000116303 s, 86.0 kB/s 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81310-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81310-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:56:12.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-03-25T00:56:12.389 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:56:12.389 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:56:12.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:56:12.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:56:12.390 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:56:13.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:56:13.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81310-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:13.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-03-25T00:56:13.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:56:13.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T00:56:13.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:56:13.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-03-25T00:56:13.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81310-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:16.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-03-25T00:56:16.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:56:16.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:56:16.729 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:56:16.730 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:56:16.731 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:56:16.731 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:56:16.731 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:56:16.731 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:56:16.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:56:16.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:56:16.731 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:56:16.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:56:16.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:56:16.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:56:16.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:56:16.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:56:16.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:56:16.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:56:16.732 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:56:16.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T00:56:16.733 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.1 2024-03-25T00:56:16.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:56:16.733 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-03-25T00:56:16.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T00:56:16.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:56:16.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:56:16.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:56:16.765 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:16.768+0000 7f46425827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:56:16.768 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:16.772+0000 7f46425827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:56:16.769 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:16.772+0000 7f46425827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:56:17.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T00:56:17.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:56:17.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:56:17.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:56:17.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:56:17.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:17.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:56:17.165 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:56:17.165 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:17.166 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:56:17.336 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:17.340+0000 7f46425827c0 -1 Falling back to public interface 2024-03-25T00:56:17.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:56:18.601 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:56:18.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:18.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:56:18.602 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:56:18.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:18.602 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:56:19.003 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:56:20.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:56:20.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:20.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:56:20.005 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:56:20.006 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:20.006 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:56:20.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:56:20.451 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:20.456+0000 7f46425827c0 -1 osd.1 48 log_to_monitors true 2024-03-25T00:56:21.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:56:21.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:21.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:56:21.409 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:56:21.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:21.410 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:56:22.094 INFO:tasks.workunit.client.0.smithi080.stderr:osd.1 up in weight 1 up_from 53 up_thru 39 down_at 49 last_clean_interval [13,48) [v2:127.0.0.1:6810/3027696421,v1:127.0.0.1:6811/3027696421] [v2:127.0.0.1:6812/3027696421,v1:127.0.0.1:6813/3027696421] exists,up 8e4d9e43-8bdb-4aca-b79c-aeeb9131c594 2024-03-25T00:56:22.094 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:56:22.094 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:56:22.094 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:56:22.094 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:56:22.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:56:22.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:56:22.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:56:22.096 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:56:22.096 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:56:22.096 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:56:22.096 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:56:22.096 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:56:22.096 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:56:22.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:56:22.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:56:22.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:56:22.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:56:22.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:56:22.205 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:56:22.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:56:22.959 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:56:22.959 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:56:22.959 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T00:56:22.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:56:22.959 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:22.959 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:56:23.378 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803822 2024-03-25T00:56:23.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803822 2024-03-25T00:56:23.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803822' 2024-03-25T00:56:23.379 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:23.380 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:56:23.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266691 2024-03-25T00:56:23.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266691 2024-03-25T00:56:23.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803822 1-227633266691' 2024-03-25T00:56:23.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:23.598 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:56:23.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378654 2024-03-25T00:56:23.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378654 2024-03-25T00:56:23.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803822 1-227633266691 2-81604378654' 2024-03-25T00:56:23.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:23.874 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:56:24.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626441 2024-03-25T00:56:24.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626441 2024-03-25T00:56:24.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803822 1-227633266691 2-81604378654 3-180388626441' 2024-03-25T00:56:24.132 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:24.133 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803822 2024-03-25T00:56:24.133 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:24.134 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:56:24.135 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:24.135 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803822 2024-03-25T00:56:24.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803822 2024-03-25T00:56:24.136 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803822' 2024-03-25T00:56:24.137 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 25769803822 2024-03-25T00:56:24.137 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:56:24.541 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803820 -lt 25769803822 2024-03-25T00:56:24.541 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:56:25.543 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:56:25.543 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:56:25.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803822 -lt 25769803822 2024-03-25T00:56:25.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:25.953 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266691 2024-03-25T00:56:25.953 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:25.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:56:25.955 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266691 2024-03-25T00:56:25.955 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:25.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266691 2024-03-25T00:56:25.957 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266691' 2024-03-25T00:56:25.957 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 227633266691 2024-03-25T00:56:25.957 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:56:26.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266691 -lt 227633266691 2024-03-25T00:56:26.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:26.364 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378654 2024-03-25T00:56:26.364 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:26.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:56:26.366 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:26.366 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378654 2024-03-25T00:56:26.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378654 2024-03-25T00:56:26.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378654' 2024-03-25T00:56:26.367 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 81604378654 2024-03-25T00:56:26.368 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:56:26.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378654 -lt 81604378654 2024-03-25T00:56:26.790 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:26.791 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-180388626441 2024-03-25T00:56:26.791 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:26.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:56:26.793 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:26.793 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-180388626441 2024-03-25T00:56:26.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626441 2024-03-25T00:56:26.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 180388626441' 2024-03-25T00:56:26.795 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 180388626441 2024-03-25T00:56:26.795 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:56:27.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626442 -lt 180388626441 2024-03-25T00:56:27.237 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:56:27.237 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:56:27.238 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:56:27.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:56:27.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:56:27.814 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:56:27.814 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:56:27.814 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:56:27.814 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:56:27.814 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:56:27.814 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:56:28.201 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:56:28.202 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:56:28.202 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:56:28.202 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:56:28.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:56:28.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:56:28.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:56:28.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-03-25T00:56:28.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-03-25T00:56:29.489 INFO:tasks.workunit.client.0.smithi080.stderr:noout is unset 2024-03-25T00:56:29.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81310-0-10 fail 2024-03-25T00:56:29.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:56:29.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:56:29.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81310-0-10 2024-03-25T00:56:29.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-03-25T00:56:29.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-03-25T00:56:29.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81310-0-10 td/test-erasure-eio/COPY 2024-03-25T00:56:29.751 INFO:tasks.workunit.client.0.smithi080.stderr:error getting pool-jerasure/obj-size-81310-0-10: (5) Input/output error 2024-03-25T00:56:29.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-03-25T00:56:29.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-03-25T00:56:29.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2024-03-25T00:56:29.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-03-25T00:56:29.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-03-25T00:56:29.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-03-25T00:56:29.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-03-25T00:56:29.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2024-03-25T00:56:29.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-03-25T00:56:29.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-03-25T00:56:29.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-03-25T00:56:29.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-81310-0-0 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81310-0-0 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81310-0-0 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:56:29.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81310-0-0 td/test-erasure-eio/ORIGINAL 2024-03-25T00:56:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-81310-0-0 td/test-erasure-eio 0 0 set 2024-03-25T00:56:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81310-0-0 2024-03-25T00:56:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-03-25T00:56:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-03-25T00:56:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-03-25T00:56:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-03-25T00:56:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-03-25T00:56:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-03-25T00:56:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-03-25T00:56:29.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-03-25T00:56:29.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-03-25T00:56:29.976 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81310-0-0 2024-03-25T00:56:29.976 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:56:29.976 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81310-0-0 2024-03-25T00:56:29.976 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81310-0-0 2024-03-25T00:56:29.976 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:56:30.404 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:56:30.404 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:56:30.404 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-25T00:56:30.404 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-03-25T00:56:30.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-03-25T00:56:30.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-03-25T00:56:30.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-03-25T00:56:30.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-03-25T00:56:31.413 INFO:tasks.workunit.client.0.smithi080.stderr:noout is set 2024-03-25T00:56:31.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-03-25T00:56:31.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-03-25T00:56:31.436 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-03-25T00:56:31.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81310-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:31.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-03-25T00:56:31.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:56:31.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T00:56:31.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:56:31.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81310-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:31.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-03-25T00:56:31.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:56:31.439 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T00:56:31.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:56:31.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-03-25T00:56:31.440 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:56:31.440 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:56:31.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:56:31.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:56:31.440 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:56:32.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:56:32.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81310-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:32.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-03-25T00:56:32.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:56:32.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T00:56:32.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:56:32.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-03-25T00:56:32.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81310-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:35.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-03-25T00:56:35.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:56:35.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:56:35.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T00:56:35.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:56:35.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:56:35.138 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:56:35.139 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:56:35.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:56:35.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T00:56:35.141 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.3 2024-03-25T00:56:35.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:56:35.141 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-03-25T00:56:35.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T00:56:35.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:56:35.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:56:35.143 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:56:35.175 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:35.180+0000 7ff0cb4c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:56:35.178 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:35.184+0000 7ff0cb4c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:56:35.179 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:35.184+0000 7ff0cb4c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:56:35.561 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T00:56:35.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:56:35.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:56:35.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:56:35.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:56:35.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:35.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:56:35.562 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:56:35.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:35.562 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:56:35.735 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:35.740+0000 7ff0cb4c77c0 -1 Falling back to public interface 2024-03-25T00:56:36.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:56:37.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:56:37.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:37.012 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:56:37.012 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:56:37.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:37.013 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:56:37.449 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:56:38.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:56:38.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:38.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:56:38.451 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:56:38.451 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:38.452 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:56:38.522 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:38.528+0000 7ff0cb4c77c0 -1 osd.3 58 log_to_monitors true 2024-03-25T00:56:39.135 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:56:40.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:56:40.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:40.137 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:56:40.137 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:56:40.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:40.138 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:56:40.881 INFO:tasks.workunit.client.0.smithi080.stderr:osd.3 up in weight 1 up_from 63 up_thru 63 down_at 60 last_clean_interval [42,58) [v2:127.0.0.1:6826/2370584423,v1:127.0.0.1:6827/2370584423] [v2:127.0.0.1:6828/2370584423,v1:127.0.0.1:6829/2370584423] exists,up d7747434-5bbc-4eef-8999-bff773a9396a 2024-03-25T00:56:40.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:56:40.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:56:40.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:56:40.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:56:40.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:56:40.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:56:40.881 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:56:40.881 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:56:40.882 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:56:40.882 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:56:40.882 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:56:40.882 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:56:40.882 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:56:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:56:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:56:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:56:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:56:40.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:56:40.988 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:56:41.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:56:41.421 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:56:41.422 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:56:41.422 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T00:56:41.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:56:41.422 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:41.422 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:56:41.632 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803827 2024-03-25T00:56:41.632 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803827 2024-03-25T00:56:41.632 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803827' 2024-03-25T00:56:41.632 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:41.632 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:56:41.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266697 2024-03-25T00:56:41.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266697 2024-03-25T00:56:41.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803827 1-227633266697' 2024-03-25T00:56:41.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:41.852 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:56:42.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378660 2024-03-25T00:56:42.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378660 2024-03-25T00:56:42.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803827 1-227633266697 2-81604378660' 2024-03-25T00:56:42.009 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:42.009 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:56:42.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939651 2024-03-25T00:56:42.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939651 2024-03-25T00:56:42.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803827 1-227633266697 2-81604378660 3-270582939651' 2024-03-25T00:56:42.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:42.158 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803827 2024-03-25T00:56:42.158 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:42.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:56:42.160 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803827 2024-03-25T00:56:42.160 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:42.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803827 2024-03-25T00:56:42.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-03-25T00:56:42.161 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 25769803827 2024-03-25T00:56:42.161 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:56:42.549 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803825 -lt 25769803827 2024-03-25T00:56:42.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:56:43.550 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:56:43.552 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:56:43.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803828 -lt 25769803827 2024-03-25T00:56:43.958 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:43.959 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266697 2024-03-25T00:56:43.960 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:43.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:56:43.962 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266697 2024-03-25T00:56:43.962 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:43.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266697 2024-03-25T00:56:43.964 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266697' 2024-03-25T00:56:43.964 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 227633266697 2024-03-25T00:56:43.964 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:56:44.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266697 -lt 227633266697 2024-03-25T00:56:44.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:44.434 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378660 2024-03-25T00:56:44.434 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:44.435 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:56:44.436 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378660 2024-03-25T00:56:44.436 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:44.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378660 2024-03-25T00:56:44.438 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378660' 2024-03-25T00:56:44.438 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 81604378660 2024-03-25T00:56:44.438 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:56:44.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378660 -lt 81604378660 2024-03-25T00:56:44.873 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:56:44.873 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-270582939651 2024-03-25T00:56:44.873 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:56:44.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:56:44.875 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-270582939651 2024-03-25T00:56:44.876 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:56:44.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939651 2024-03-25T00:56:44.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 270582939651' 2024-03-25T00:56:44.877 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 270582939651 2024-03-25T00:56:44.877 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:56:45.359 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939652 -lt 270582939651 2024-03-25T00:56:45.360 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:56:45.360 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:56:45.360 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:56:45.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:56:45.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:56:45.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:56:45.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:56:45.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:56:45.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:56:45.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:56:45.911 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:56:46.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:56:46.292 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:56:46.293 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:56:46.293 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:56:46.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:56:46.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:56:46.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:56:46.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-03-25T00:56:46.846 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-03-25T00:56:47.804 INFO:tasks.workunit.client.0.smithi080.stderr:noout is unset 2024-03-25T00:56:47.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81310-0-0 2024-03-25T00:56:47.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:56:47.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:56:47.824 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81310-0-0 2024-03-25T00:56:47.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:56:47.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:56:47.825 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81310-0-0 td/test-erasure-eio/COPY 2024-03-25T00:56:48.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:56:48.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:56:48.106 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-81310-0-0 td/test-erasure-eio 1 0 set 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81310-0-0 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-03-25T00:56:48.108 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-03-25T00:56:48.109 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81310-0-0 2024-03-25T00:56:48.109 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:56:48.109 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81310-0-0 2024-03-25T00:56:48.110 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81310-0-0 2024-03-25T00:56:48.110 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:56:48.510 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:56:48.511 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:56:48.511 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-25T00:56:48.511 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-03-25T00:56:48.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-03-25T00:56:48.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-03-25T00:56:48.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-03-25T00:56:48.511 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-03-25T00:56:49.343 INFO:tasks.workunit.client.0.smithi080.stderr:noout is set 2024-03-25T00:56:49.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-03-25T00:56:49.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-03-25T00:56:49.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-03-25T00:56:49.365 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81310-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:49.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-03-25T00:56:49.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:56:49.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T00:56:49.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:56:49.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81310-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:49.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-03-25T00:56:49.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:56:49.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T00:56:49.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:56:49.366 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-03-25T00:56:49.366 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:56:49.367 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:56:49.367 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:56:49.369 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:56:49.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:56:50.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:56:50.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81310-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:50.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-03-25T00:56:50.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:56:50.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T00:56:50.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:56:50.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-03-25T00:56:50.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81310-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:56:53.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-03-25T00:56:53.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:56:53.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:56:53.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T00:56:53.245 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:56:53.246 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:56:53.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:56:53.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T00:56:53.249 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.1 2024-03-25T00:56:53.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:56:53.249 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-03-25T00:56:53.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T00:56:53.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:56:53.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:56:53.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:56:53.281 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:53.283+0000 7f54ae9667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:56:53.284 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:53.287+0000 7f54ae9667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:56:53.285 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:53.291+0000 7f54ae9667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:56:53.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T00:56:53.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:56:53.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:56:53.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:56:53.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:56:53.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:53.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:56:53.692 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:56:53.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:53.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:56:53.858 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:53.863+0000 7f54ae9667c0 -1 Falling back to public interface 2024-03-25T00:56:54.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:56:55.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:56:55.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:55.094 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:56:55.094 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:56:55.094 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:55.094 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:56:55.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:56:56.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:56:56.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:56.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:56:56.495 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:56:56.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:56.495 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:56:56.555 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:56:56.559+0000 7f54ae9667c0 -1 osd.1 68 log_to_monitors true 2024-03-25T00:56:57.006 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:56:58.007 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:56:58.007 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:56:58.007 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:56:58.007 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:56:58.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:56:58.008 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:56:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:osd.1 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [53,68) [v2:127.0.0.1:6810/2869828414,v1:127.0.0.1:6811/2869828414] [v2:127.0.0.1:6812/2869828414,v1:127.0.0.1:6813/2869828414] exists,up 8e4d9e43-8bdb-4aca-b79c-aeeb9131c594 2024-03-25T00:56:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:56:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:56:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:56:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:56:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:56:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:56:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:56:58.583 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:56:58.583 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:56:58.584 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:56:58.584 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:56:58.584 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:56:58.584 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:56:58.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:56:58.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:56:58.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:56:58.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:56:58.692 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:56:58.692 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:56:59.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:56:59.299 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:56:59.300 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:56:59.300 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T00:56:59.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:56:59.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:59.300 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:56:59.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803833 2024-03-25T00:56:59.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803833 2024-03-25T00:56:59.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803833' 2024-03-25T00:56:59.649 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:59.649 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:56:59.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579907 2024-03-25T00:56:59.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579907 2024-03-25T00:56:59.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803833 1-317827579907' 2024-03-25T00:56:59.814 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:56:59.814 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:57:00.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378666 2024-03-25T00:57:00.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378666 2024-03-25T00:57:00.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803833 1-317827579907 2-81604378666' 2024-03-25T00:57:00.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:00.002 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:57:00.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939657 2024-03-25T00:57:00.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939657 2024-03-25T00:57:00.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803833 1-317827579907 2-81604378666 3-270582939657' 2024-03-25T00:57:00.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:00.178 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803833 2024-03-25T00:57:00.178 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:00.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:57:00.180 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803833 2024-03-25T00:57:00.180 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:00.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803833 2024-03-25T00:57:00.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803833' 2024-03-25T00:57:00.181 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 25769803833 2024-03-25T00:57:00.181 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:57:00.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803831 -lt 25769803833 2024-03-25T00:57:00.597 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:57:01.598 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:57:01.599 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:57:02.025 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803833 -lt 25769803833 2024-03-25T00:57:02.025 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:02.025 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579907 2024-03-25T00:57:02.026 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:02.027 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:57:02.028 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579907 2024-03-25T00:57:02.028 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:02.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579907 2024-03-25T00:57:02.029 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579907' 2024-03-25T00:57:02.029 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 317827579907 2024-03-25T00:57:02.029 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:57:02.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579907 -lt 317827579907 2024-03-25T00:57:02.462 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:02.463 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:02.463 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378666 2024-03-25T00:57:02.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:57:02.465 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378666 2024-03-25T00:57:02.465 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:02.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378666 2024-03-25T00:57:02.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378666' 2024-03-25T00:57:02.467 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 81604378666 2024-03-25T00:57:02.467 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:57:02.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378666 -lt 81604378666 2024-03-25T00:57:02.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:02.932 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-270582939657 2024-03-25T00:57:02.933 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:02.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:57:02.935 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-270582939657 2024-03-25T00:57:02.935 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:02.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939657 2024-03-25T00:57:02.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 270582939657' 2024-03-25T00:57:02.937 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 270582939657 2024-03-25T00:57:02.937 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:57:03.370 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939657 -lt 270582939657 2024-03-25T00:57:03.371 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:57:03.372 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:57:03.372 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:57:03.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:57:03.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:57:03.976 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:57:03.976 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:57:03.976 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:57:03.976 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:57:03.977 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:57:03.977 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:57:04.359 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:57:04.359 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:57:04.360 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:57:04.360 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:57:04.893 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:57:04.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:57:04.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:57:04.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-03-25T00:57:04.894 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-03-25T00:57:05.736 INFO:tasks.workunit.client.0.smithi080.stderr:noout is unset 2024-03-25T00:57:05.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81310-0-0 fail 2024-03-25T00:57:05.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:57:05.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:57:05.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81310-0-0 2024-03-25T00:57:05.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-03-25T00:57:05.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-03-25T00:57:05.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81310-0-0 td/test-erasure-eio/COPY 2024-03-25T00:57:05.949 INFO:tasks.workunit.client.0.smithi080.stderr:error getting pool-jerasure/obj-size-81310-0-0: (5) Input/output error 2024-03-25T00:57:05.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-03-25T00:57:05.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-03-25T00:57:05.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:311: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2024-03-25T00:57:05.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-03-25T00:57:05.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-03-25T00:57:05.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-03-25T00:57:05.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-03-25T00:57:05.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2024-03-25T00:57:05.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-03-25T00:57:05.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-81310-0-256 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81310-0-256 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81310-0-256 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T00:57:05.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81310-0-256 td/test-erasure-eio/ORIGINAL 2024-03-25T00:57:06.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-81310-0-256 td/test-erasure-eio 0 256 add 2024-03-25T00:57:06.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81310-0-256 2024-03-25T00:57:06.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-03-25T00:57:06.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-03-25T00:57:06.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-03-25T00:57:06.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-03-25T00:57:06.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-03-25T00:57:06.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-03-25T00:57:06.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-03-25T00:57:06.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-03-25T00:57:06.269 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-03-25T00:57:06.270 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81310-0-256 2024-03-25T00:57:06.270 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:57:06.270 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81310-0-256 2024-03-25T00:57:06.270 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81310-0-256 2024-03-25T00:57:06.270 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:57:06.769 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:57:06.769 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:57:06.769 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-25T00:57:06.769 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-03-25T00:57:06.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-03-25T00:57:06.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-03-25T00:57:06.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-03-25T00:57:06.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-03-25T00:57:07.685 INFO:tasks.workunit.client.0.smithi080.stderr:noout is set 2024-03-25T00:57:07.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-03-25T00:57:07.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81310-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:07.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-03-25T00:57:07.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:57:07.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T00:57:07.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:57:07.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81310-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:07.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-03-25T00:57:07.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:57:07.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T00:57:07.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:57:07.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-03-25T00:57:07.711 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:57:07.711 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:57:07.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:57:07.711 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:57:07.712 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:57:09.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:57:09.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81310-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:09.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-03-25T00:57:09.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:57:09.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T00:57:09.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:57:09.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-03-25T00:57:09.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81310-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:11.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-03-25T00:57:11.172 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:57:11.173 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:57:11.174 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:57:11.174 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:57:11.175 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:57:11.175 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:57:11.175 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:57:11.175 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:57:11.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:57:11.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:57:11.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:57:11.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:57:11.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:57:11.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:57:11.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:57:11.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:57:11.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:57:11.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:57:11.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:57:11.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T00:57:11.176 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.3 2024-03-25T00:57:11.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:57:11.177 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-03-25T00:57:11.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T00:57:11.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:57:11.178 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:57:11.179 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:57:11.216 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:11.219+0000 7f199a5277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:57:11.218 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:11.223+0000 7f199a5277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:57:11.220 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:11.223+0000 7f199a5277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:57:11.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T00:57:11.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:57:11.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:57:11.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:57:11.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:57:11.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:11.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:57:11.594 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:57:11.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:57:11.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:11.766 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:11.771+0000 7f199a5277c0 -1 Falling back to public interface 2024-03-25T00:57:12.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:57:13.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:57:13.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:13.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:57:13.041 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:57:13.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:13.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:57:13.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:57:14.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:57:14.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:14.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:57:14.493 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:57:14.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:14.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:57:14.549 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:14.555+0000 7f199a5277c0 -1 osd.3 81 log_to_monitors true 2024-03-25T00:57:15.302 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:57:16.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:57:16.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:16.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:57:16.305 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:57:16.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:16.305 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:57:16.720 INFO:tasks.workunit.client.0.smithi080.stderr:osd.3 up in weight 1 up_from 85 up_thru 85 down_at 82 last_clean_interval [63,81) [v2:127.0.0.1:6826/3925983910,v1:127.0.0.1:6827/3925983910] [v2:127.0.0.1:6828/3925983910,v1:127.0.0.1:6829/3925983910] exists,up d7747434-5bbc-4eef-8999-bff773a9396a 2024-03-25T00:57:16.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:57:16.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:57:16.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:57:16.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:57:16.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:57:16.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:57:16.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:57:16.721 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:57:16.722 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:57:16.722 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:57:16.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:57:16.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:57:16.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:57:16.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:57:16.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:57:16.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:57:16.829 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:57:16.830 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:57:16.830 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:57:17.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:57:17.586 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:57:17.586 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:57:17.586 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T00:57:17.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:57:17.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:17.586 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:57:17.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803838 2024-03-25T00:57:17.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803838 2024-03-25T00:57:17.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838' 2024-03-25T00:57:17.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:17.744 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:57:17.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579913 2024-03-25T00:57:17.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579913 2024-03-25T00:57:17.993 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-317827579913' 2024-03-25T00:57:17.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:17.994 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:57:18.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378671 2024-03-25T00:57:18.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378671 2024-03-25T00:57:18.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-317827579913 2-81604378671' 2024-03-25T00:57:18.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:18.149 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:57:18.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220163 2024-03-25T00:57:18.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220163 2024-03-25T00:57:18.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-317827579913 2-81604378671 3-365072220163' 2024-03-25T00:57:18.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:18.300 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803838 2024-03-25T00:57:18.300 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:18.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:57:18.302 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803838 2024-03-25T00:57:18.302 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:18.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803838 2024-03-25T00:57:18.304 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803838' 2024-03-25T00:57:18.304 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 25769803838 2024-03-25T00:57:18.304 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:57:18.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803836 -lt 25769803838 2024-03-25T00:57:18.727 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:57:19.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:57:19.730 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:57:20.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803839 -lt 25769803838 2024-03-25T00:57:20.218 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:20.218 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579913 2024-03-25T00:57:20.219 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:20.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:57:20.221 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579913 2024-03-25T00:57:20.221 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:20.222 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579913 2024-03-25T00:57:20.223 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579913' 2024-03-25T00:57:20.223 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 317827579913 2024-03-25T00:57:20.223 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:57:20.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579913 -lt 317827579913 2024-03-25T00:57:20.673 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:20.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:20.674 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378671 2024-03-25T00:57:20.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:57:20.676 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378671 2024-03-25T00:57:20.676 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:20.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378671 2024-03-25T00:57:20.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378671' 2024-03-25T00:57:20.677 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 81604378671 2024-03-25T00:57:20.677 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:57:21.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378672 -lt 81604378671 2024-03-25T00:57:21.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:21.126 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-365072220163 2024-03-25T00:57:21.126 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:21.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:57:21.130 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:21.130 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-365072220163 2024-03-25T00:57:21.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220163 2024-03-25T00:57:21.130 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 365072220163' 2024-03-25T00:57:21.131 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 365072220163 2024-03-25T00:57:21.131 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:57:21.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220164 -lt 365072220163 2024-03-25T00:57:21.518 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:57:21.519 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:57:21.519 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:57:22.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:57:22.200 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:57:22.201 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:57:22.201 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:57:22.201 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:57:22.201 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:57:22.201 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:57:22.202 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:57:22.609 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:57:22.610 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:57:22.610 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:57:22.610 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:57:23.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:57:23.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:57:23.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:57:23.150 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-03-25T00:57:23.152 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records in 2024-03-25T00:57:23.152 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records out 2024-03-25T00:57:23.152 INFO:tasks.workunit.client.0.smithi080.stderr:256 bytes copied, 3.5496e-05 s, 7.2 MB/s 2024-03-25T00:57:23.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81310-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:23.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-03-25T00:57:23.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:57:23.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-25T00:57:23.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:57:23.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81310-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:23.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-03-25T00:57:23.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:57:23.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-25T00:57:23.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:57:23.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-03-25T00:57:23.153 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:57:23.153 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:57:23.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:57:23.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:57:23.153 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:57:23.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:57:23.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81310-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:23.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-03-25T00:57:23.459 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:57:23.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-25T00:57:23.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:57:23.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-03-25T00:57:23.460 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81310-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:26.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-03-25T00:57:26.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:57:26.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:57:26.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-25T00:57:26.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:57:26.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T00:57:26.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:57:26.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:57:26.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:57:26.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:57:26.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T00:57:26.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T00:57:26.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:57:26.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:57:26.265 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:57:26.266 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:57:26.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:57:26.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:57:26.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:57:26.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:57:26.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:57:26.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T00:57:26.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-25T00:57:26.268 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.3 2024-03-25T00:57:26.268 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:57:26.269 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-03-25T00:57:26.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-25T00:57:26.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:57:26.270 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:57:26.271 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:57:26.308 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:26.310+0000 7ff7aed457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:57:26.310 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:26.314+0000 7ff7aed457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:57:26.311 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:26.314+0000 7ff7aed457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:57:26.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-25T00:57:26.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:57:26.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T00:57:26.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:57:26.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:57:26.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:26.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:57:26.685 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:57:26.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:26.685 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:57:26.863 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:26.866+0000 7ff7aed457c0 -1 Falling back to public interface 2024-03-25T00:57:27.081 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:57:28.082 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:57:28.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:28.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:57:28.083 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:57:28.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:28.083 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:57:28.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:57:29.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:57:29.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:29.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:57:29.505 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:57:29.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:29.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:57:30.070 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:30.074+0000 7ff7aed457c0 -1 osd.3 87 log_to_monitors true 2024-03-25T00:57:30.111 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:57:31.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:57:31.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:31.113 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:57:31.113 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:57:31.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:31.114 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T00:57:31.532 INFO:tasks.workunit.client.0.smithi080.stderr:osd.3 up in weight 1 up_from 92 up_thru 92 down_at 88 last_clean_interval [85,87) [v2:127.0.0.1:6826/990803490,v1:127.0.0.1:6827/990803490] [v2:127.0.0.1:6828/990803490,v1:127.0.0.1:6829/990803490] exists,up d7747434-5bbc-4eef-8999-bff773a9396a 2024-03-25T00:57:31.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:57:31.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:57:31.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:57:31.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:57:31.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:57:31.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:57:31.532 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:57:31.533 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:57:31.533 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:57:31.533 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:57:31.534 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:57:31.534 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:57:31.534 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:57:31.633 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:57:31.633 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:57:31.633 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:57:31.633 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:57:31.633 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:57:31.634 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:57:32.486 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:57:32.487 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:57:32.487 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:57:32.487 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T00:57:32.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:57:32.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:32.487 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:57:32.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803843 2024-03-25T00:57:32.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803843 2024-03-25T00:57:32.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843' 2024-03-25T00:57:32.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:32.837 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:57:33.064 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579918 2024-03-25T00:57:33.064 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579918 2024-03-25T00:57:33.064 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-317827579918' 2024-03-25T00:57:33.064 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:33.064 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:57:33.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378676 2024-03-25T00:57:33.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378676 2024-03-25T00:57:33.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-317827579918 2-81604378676' 2024-03-25T00:57:33.219 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:33.219 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:57:33.498 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991235 2024-03-25T00:57:33.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991235 2024-03-25T00:57:33.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803843 1-317827579918 2-81604378676 3-395136991235' 2024-03-25T00:57:33.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:33.499 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:33.499 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803843 2024-03-25T00:57:33.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:57:33.501 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803843 2024-03-25T00:57:33.501 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:33.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803843 2024-03-25T00:57:33.503 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803843' 2024-03-25T00:57:33.503 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 25769803843 2024-03-25T00:57:33.503 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:57:33.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803844 -lt 25769803843 2024-03-25T00:57:33.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:33.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-317827579918 2024-03-25T00:57:33.928 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:33.929 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:57:33.930 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-317827579918 2024-03-25T00:57:33.930 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:33.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579918 2024-03-25T00:57:33.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 317827579918' 2024-03-25T00:57:33.931 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 317827579918 2024-03-25T00:57:33.932 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:57:34.330 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579916 -lt 317827579918 2024-03-25T00:57:34.331 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:57:35.332 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:57:35.332 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:57:35.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579918 -lt 317827579918 2024-03-25T00:57:35.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:35.795 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378676 2024-03-25T00:57:35.796 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:35.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:57:35.797 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378676 2024-03-25T00:57:35.798 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:35.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378676 2024-03-25T00:57:35.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378676' 2024-03-25T00:57:35.799 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 81604378676 2024-03-25T00:57:35.800 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:57:36.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378677 -lt 81604378676 2024-03-25T00:57:36.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:36.196 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991235 2024-03-25T00:57:36.196 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:36.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:57:36.198 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:36.198 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991235 2024-03-25T00:57:36.199 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991235 2024-03-25T00:57:36.200 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991235' 2024-03-25T00:57:36.200 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 395136991235 2024-03-25T00:57:36.200 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:57:36.627 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991235 -lt 395136991235 2024-03-25T00:57:36.628 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:57:36.629 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:57:36.629 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:57:37.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:57:37.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:57:37.229 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:57:37.229 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:57:37.229 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:57:37.229 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:57:37.229 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:57:37.229 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:57:37.635 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T00:57:37.635 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:57:37.636 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:57:37.636 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:57:38.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T00:57:38.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T00:57:38.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T00:57:38.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T00:57:38.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T00:57:38.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T00:57:38.263 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T00:57:38.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:57:38.264 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:57:38.264 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:57:38.264 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:57:38.265 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:57:38.265 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:57:38.265 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:57:38.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T00:57:38.675 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:57:38.676 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:57:38.676 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:57:39.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T00:57:39.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-25T00:57:39.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-25T00:57:39.225 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-25T00:57:39.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-25T00:57:39.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-25T00:57:39.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-25T00:57:39.226 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-25T00:57:39.226 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-25T00:57:39.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-25T00:57:39.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-25T00:57:39.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-25T00:57:39.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T00:57:39.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-25T00:57:39.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T00:57:39.961 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:57:39.962 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:57:39.962 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:57:39.962 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:57:39.962 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:57:39.963 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:57:39.963 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:57:40.383 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:57:40.384 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:57:40.384 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:57:40.384 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:57:40.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:57:40.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:57:40.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:57:40.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-03-25T00:57:40.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-03-25T00:57:41.795 INFO:tasks.workunit.client.0.smithi080.stderr:noout is unset 2024-03-25T00:57:41.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81310-0-256 2024-03-25T00:57:41.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:57:41.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:57:41.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81310-0-256 2024-03-25T00:57:41.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T00:57:41.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T00:57:41.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81310-0-256 td/test-erasure-eio/COPY 2024-03-25T00:57:42.140 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T00:57:42.141 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T00:57:42.180 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-03-25T00:57:42.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-03-25T00:57:42.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-81310-0-256 td/test-erasure-eio 1 256 add 2024-03-25T00:57:42.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81310-0-256 2024-03-25T00:57:42.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-03-25T00:57:42.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-03-25T00:57:42.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-03-25T00:57:42.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-03-25T00:57:42.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-03-25T00:57:42.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-03-25T00:57:42.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-03-25T00:57:42.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-03-25T00:57:42.182 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-03-25T00:57:42.183 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81310-0-256 2024-03-25T00:57:42.183 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T00:57:42.183 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81310-0-256 2024-03-25T00:57:42.183 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81310-0-256 2024-03-25T00:57:42.184 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T00:57:42.606 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T00:57:42.606 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:57:42.606 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-25T00:57:42.606 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-03-25T00:57:42.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-03-25T00:57:42.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-03-25T00:57:42.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-03-25T00:57:42.607 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-03-25T00:57:43.451 INFO:tasks.workunit.client.0.smithi080.stderr:noout is set 2024-03-25T00:57:43.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-03-25T00:57:43.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81310-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:43.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-03-25T00:57:43.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:57:43.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T00:57:43.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:57:43.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81310-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:43.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-03-25T00:57:43.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:57:43.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T00:57:43.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:57:43.473 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-03-25T00:57:43.473 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:57:43.474 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:57:43.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:57:43.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:57:43.474 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:57:44.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:57:44.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81310-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:44.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-03-25T00:57:44.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:57:44.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T00:57:44.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:57:44.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-03-25T00:57:44.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81310-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:47.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-03-25T00:57:47.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:57:47.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:57:47.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T00:57:47.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:57:47.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:57:47.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:57:47.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:57:47.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:57:47.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:57:47.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:57:47.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:57:47.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:57:47.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:57:47.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:57:47.045 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:57:47.045 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:57:47.045 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:57:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:57:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:57:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:57:47.045 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:57:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:57:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:57:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:57:47.045 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:57:47.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:57:47.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:57:47.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:57:47.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:57:47.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:57:47.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:57:47.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:57:47.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T00:57:47.047 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.1 2024-03-25T00:57:47.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:57:47.047 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-03-25T00:57:47.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T00:57:47.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:57:47.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:57:47.048 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:57:47.080 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:47.086+0000 7f937d3d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:57:47.082 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:47.086+0000 7f937d3d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:57:47.084 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:47.090+0000 7f937d3d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:57:47.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T00:57:47.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:57:47.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:57:47.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:57:47.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:57:47.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:47.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:57:47.509 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:57:47.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:47.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:57:47.625 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:47.630+0000 7f937d3d27c0 -1 Falling back to public interface 2024-03-25T00:57:47.922 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:57:48.923 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:57:48.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:48.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:57:48.924 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:57:48.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:48.924 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:57:49.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:57:50.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:57:50.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:50.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:57:50.349 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:57:50.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:50.350 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:57:50.365 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:57:50.370+0000 7f937d3d27c0 -1 osd.1 96 log_to_monitors true 2024-03-25T00:57:50.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:57:51.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:57:51.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:57:51.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:57:51.870 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:57:51.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:57:51.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:57:52.701 INFO:tasks.workunit.client.0.smithi080.stderr:osd.1 up in weight 1 up_from 102 up_thru 102 down_at 99 last_clean_interval [74,96) [v2:127.0.0.1:6810/2435042300,v1:127.0.0.1:6811/2435042300] [v2:127.0.0.1:6812/2435042300,v1:127.0.0.1:6813/2435042300] exists,up 8e4d9e43-8bdb-4aca-b79c-aeeb9131c594 2024-03-25T00:57:52.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:57:52.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:57:52.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:57:52.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:57:52.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:57:52.701 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:57:52.702 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:57:52.702 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:57:52.702 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:57:52.702 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:57:52.703 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:57:52.703 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:57:52.703 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:57:52.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:57:52.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:57:52.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:57:52.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:57:52.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:57:52.816 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:57:53.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:57:53.689 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:57:53.689 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:57:53.690 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T00:57:53.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:57:53.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:53.690 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:57:53.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803850 2024-03-25T00:57:53.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803850 2024-03-25T00:57:53.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850' 2024-03-25T00:57:53.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:53.851 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:57:54.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=438086664195 2024-03-25T00:57:54.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 438086664195 2024-03-25T00:57:54.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-438086664195' 2024-03-25T00:57:54.002 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:54.003 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:57:54.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378682 2024-03-25T00:57:54.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378682 2024-03-25T00:57:54.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-438086664195 2-81604378682' 2024-03-25T00:57:54.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:57:54.158 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:57:54.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991241 2024-03-25T00:57:54.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991241 2024-03-25T00:57:54.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-438086664195 2-81604378682 3-395136991241' 2024-03-25T00:57:54.311 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:54.311 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803850 2024-03-25T00:57:54.311 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:54.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:57:54.312 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803850 2024-03-25T00:57:54.313 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:54.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803850 2024-03-25T00:57:54.314 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803850' 2024-03-25T00:57:54.314 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 25769803850 2024-03-25T00:57:54.314 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:57:54.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803848 -lt 25769803850 2024-03-25T00:57:54.762 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:57:55.764 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:57:55.765 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:57:56.175 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803850 -lt 25769803850 2024-03-25T00:57:56.176 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:56.176 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:56.176 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-438086664195 2024-03-25T00:57:56.177 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:57:56.178 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-438086664195 2024-03-25T00:57:56.178 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:56.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=438086664195 2024-03-25T00:57:56.180 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 438086664195' 2024-03-25T00:57:56.180 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 438086664195 2024-03-25T00:57:56.180 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:57:56.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 438086664195 -lt 438086664195 2024-03-25T00:57:56.580 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:56.581 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:56.581 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378682 2024-03-25T00:57:56.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:57:56.583 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378682 2024-03-25T00:57:56.583 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:56.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378682 2024-03-25T00:57:56.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378682' 2024-03-25T00:57:56.585 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 81604378682 2024-03-25T00:57:56.585 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:57:57.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378683 -lt 81604378682 2024-03-25T00:57:57.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:57:57.037 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:57:57.038 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991241 2024-03-25T00:57:57.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:57:57.039 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991241 2024-03-25T00:57:57.039 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:57:57.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991241 2024-03-25T00:57:57.041 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991241' 2024-03-25T00:57:57.041 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 395136991241 2024-03-25T00:57:57.041 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:57:57.429 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991242 -lt 395136991241 2024-03-25T00:57:57.430 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:57:57.430 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:57:57.430 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:57:57.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:57:57.960 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:57:57.960 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:57:57.960 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:57:57.960 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:57:57.960 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:57:57.961 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:57:57.961 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:57:58.343 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T00:57:58.343 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:57:58.344 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:57:58.344 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:57:58.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T00:57:58.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T00:57:58.903 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T00:57:58.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T00:57:58.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T00:57:58.904 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T00:57:59.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T00:57:59.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:57:59.006 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:57:59.006 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:57:59.006 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:57:59.006 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:57:59.006 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:57:59.010 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:57:59.413 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:57:59.414 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:57:59.418 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:57:59.418 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:57:59.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:57:59.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:57:59.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:57:59.973 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-03-25T00:57:59.974 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records in 2024-03-25T00:57:59.974 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records out 2024-03-25T00:57:59.974 INFO:tasks.workunit.client.0.smithi080.stderr:256 bytes copied, 3.4062e-05 s, 7.5 MB/s 2024-03-25T00:57:59.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81310-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:59.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-03-25T00:57:59.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T00:57:59.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T00:57:59.974 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T00:57:59.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81310-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:57:59.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-03-25T00:57:59.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T00:57:59.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T00:57:59.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T00:57:59.975 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-03-25T00:57:59.975 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:57:59.975 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:57:59.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:57:59.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:57:59.976 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:58:00.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:58:00.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81310-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:58:00.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-03-25T00:58:00.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T00:58:00.286 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T00:58:00.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T00:58:00.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-03-25T00:58:00.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81310-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T00:58:03.247 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-03-25T00:58:03.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T00:58:03.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T00:58:03.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T00:58:03.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T00:58:03.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:58:03.248 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:58:03.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:58:03.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:58:03.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:58:03.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:58:03.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:58:03.249 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T00:58:03.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T00:58:03.250 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:58:03.250 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T00:58:03.250 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:58:03.250 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:58:03.250 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:58:03.251 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:58:03.251 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:58:03.251 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:58:03.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:58:03.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:58:03.251 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:58:03.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:58:03.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:58:03.252 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:58:03.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:58:03.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:58:03.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T00:58:03.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T00:58:03.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:58:03.253 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T00:58:03.254 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.1 2024-03-25T00:58:03.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:58:03.254 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-03-25T00:58:03.254 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T00:58:03.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T00:58:03.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T00:58:03.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T00:58:03.280 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:03.286+0000 7f7a829607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:58:03.282 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:03.286+0000 7f7a829607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:58:03.284 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:03.290+0000 7f7a829607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:58:03.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T00:58:03.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:58:03.675 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:58:03.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:58:03.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:58:03.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:03.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:58:03.676 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T00:58:03.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:03.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:58:03.853 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:03.858+0000 7f7a829607c0 -1 Falling back to public interface 2024-03-25T00:58:04.123 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:05.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:05.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:05.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:58:05.125 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:58:05.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:05.125 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:58:05.593 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:06.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:06.594 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:06.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:58:06.595 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:58:06.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:06.595 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:58:06.734 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:06.738+0000 7f7a829607c0 -1 osd.1 104 log_to_monitors true 2024-03-25T00:58:07.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:08.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:08.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:08.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:58:08.080 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T00:58:08.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:08.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:58:08.799 INFO:tasks.workunit.client.0.smithi080.stderr:osd.1 up in weight 1 up_from 109 up_thru 109 down_at 105 last_clean_interval [102,104) [v2:127.0.0.1:6810/3534078992,v1:127.0.0.1:6811/3534078992] [v2:127.0.0.1:6812/3534078992,v1:127.0.0.1:6813/3534078992] exists,up 8e4d9e43-8bdb-4aca-b79c-aeeb9131c594 2024-03-25T00:58:08.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:58:08.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:58:08.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:58:08.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T00:58:08.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T00:58:08.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T00:58:08.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T00:58:08.800 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T00:58:08.800 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T00:58:08.800 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T00:58:08.801 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T00:58:08.801 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T00:58:08.801 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T00:58:08.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T00:58:08.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T00:58:08.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T00:58:08.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T00:58:08.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T00:58:08.918 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T00:58:09.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T00:58:09.623 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T00:58:09.623 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T00:58:09.624 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T00:58:09.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T00:58:09.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:58:09.624 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T00:58:09.794 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803855 2024-03-25T00:58:09.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803855 2024-03-25T00:58:09.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803855' 2024-03-25T00:58:09.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:58:09.795 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T00:58:09.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=468151435267 2024-03-25T00:58:09.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 468151435267 2024-03-25T00:58:09.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803855 1-468151435267' 2024-03-25T00:58:09.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:58:09.987 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T00:58:10.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378688 2024-03-25T00:58:10.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378688 2024-03-25T00:58:10.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803855 1-468151435267 2-81604378688' 2024-03-25T00:58:10.144 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T00:58:10.144 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T00:58:10.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991246 2024-03-25T00:58:10.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991246 2024-03-25T00:58:10.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803855 1-468151435267 2-81604378688 3-395136991246' 2024-03-25T00:58:10.290 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:58:10.290 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803855 2024-03-25T00:58:10.290 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:58:10.291 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T00:58:10.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803855 2024-03-25T00:58:10.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:58:10.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803855 2024-03-25T00:58:10.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803855' 2024-03-25T00:58:10.292 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 25769803855 2024-03-25T00:58:10.293 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:58:10.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803853 -lt 25769803855 2024-03-25T00:58:10.680 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T00:58:11.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T00:58:11.682 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T00:58:12.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803855 -lt 25769803855 2024-03-25T00:58:12.088 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:58:12.089 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-468151435267 2024-03-25T00:58:12.089 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:58:12.090 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T00:58:12.092 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-468151435267 2024-03-25T00:58:12.092 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:58:12.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=468151435267 2024-03-25T00:58:12.093 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 468151435267' 2024-03-25T00:58:12.093 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 468151435267 2024-03-25T00:58:12.093 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T00:58:12.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 468151435267 -lt 468151435267 2024-03-25T00:58:12.523 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:58:12.523 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378688 2024-03-25T00:58:12.523 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:58:12.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T00:58:12.526 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378688 2024-03-25T00:58:12.526 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:58:12.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378688 2024-03-25T00:58:12.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378688' 2024-03-25T00:58:12.527 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 81604378688 2024-03-25T00:58:12.527 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T00:58:12.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378688 -lt 81604378688 2024-03-25T00:58:12.927 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T00:58:12.929 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991246 2024-03-25T00:58:12.929 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T00:58:12.930 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T00:58:12.930 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T00:58:12.930 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991246 2024-03-25T00:58:12.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991246 2024-03-25T00:58:12.934 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991246' 2024-03-25T00:58:12.934 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 395136991246 2024-03-25T00:58:12.934 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T00:58:13.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991247 -lt 395136991246 2024-03-25T00:58:13.342 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T00:58:13.343 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:58:13.343 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:58:13.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T00:58:13.926 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T00:58:13.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T00:58:13.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T00:58:13.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T00:58:13.927 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T00:58:13.942 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T00:58:13.942 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T00:58:14.303 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T00:58:14.304 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T00:58:14.304 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T00:58:14.305 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T00:58:14.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T00:58:14.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T00:58:14.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T00:58:14.817 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-03-25T00:58:14.818 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-03-25T00:58:15.711 INFO:tasks.workunit.client.0.smithi080.stderr:noout is unset 2024-03-25T00:58:15.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81310-0-256 fail 2024-03-25T00:58:15.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T00:58:15.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T00:58:15.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81310-0-256 2024-03-25T00:58:15.733 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-03-25T00:58:15.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-03-25T00:58:15.734 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81310-0-256 td/test-erasure-eio/COPY 2024-03-25T00:58:16.012 INFO:tasks.workunit.client.0.smithi080.stderr:error getting pool-jerasure/obj-size-81310-0-256: (5) Input/output error 2024-03-25T00:58:16.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-03-25T00:58:16.017 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-03-25T00:58:16.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:312: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2024-03-25T00:58:16.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T00:58:16.018 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-03-25T00:58:16.614 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' does not exist 2024-03-25T00:58:16.629 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-03-25T00:58:17.467 INFO:tasks.workunit.client.0.smithi080.stderr:erasure-code-profile myprofile does not exist 2024-03-25T00:58:17.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-03-25T00:58:17.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:58:17.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:58:17.491 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:58:17.492 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:58:17.492 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:58:17.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:58:17.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:58:17.492 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:58:17.806 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:58:17.807 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:58:17.808 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:58:17.809 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:58:17.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:58:17.811 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:58:17.811 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:58:17.812 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:58:17.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:58:17.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:58:17.813 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:58:17.815 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:58:17.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:58:17.816 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:58:17.854 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:58:17.855 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:58:17.855 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:58:17.855 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:58:17.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:58:17.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:58:17.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-03-25T00:58:17.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-03-25T00:58:17.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-03-25T00:58:17.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-03-25T00:58:17.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T00:58:17.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-25T00:58:17.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T00:58:17.857 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T00:58:17.858 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T00:58:17.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T00:58:17.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T00:58:17.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T00:58:17.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T00:58:17.860 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T00:58:17.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T00:58:17.862 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T00:58:17.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T00:58:17.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T00:58:17.863 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T00:58:17.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:58:17.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T00:58:17.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T00:58:17.865 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T00:58:17.866 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T00:58:17.867 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-25T00:58:17.868 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T00:58:17.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T00:58:17.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:58:17.868 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:58:17.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T00:58:17.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T00:58:17.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T00:58:17.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-03-25T00:58:17.872 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-25T00:58:17.872 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:58:17.872 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:58:17.872 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81310 2024-03-25T00:58:17.874 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-25T00:58:17.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-25T00:58:17.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-25T00:58:17.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-03-25T00:58:17.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-03-25T00:58:17.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-03-25T00:58:17.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-25T00:58:17.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-25T00:58:17.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-25T00:58:17.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-03-25T00:58:17.875 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-03-25T00:58:18.413 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-25T00:58:18.413 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:58:18.413 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:58:18.414 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:58:18.414 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:58:18.414 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:58:18.415 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:58:18.415 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-25T00:58:18.648 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-25T00:58:18.649 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-25T00:58:18.650 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:58:18.650 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:58:18.650 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-25T00:58:18.650 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-25T00:58:18.651 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:58:18.651 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:58:18.651 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:58:18.652 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:58:18.652 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:58:18.652 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:58:18.652 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:58:18.653 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:58:18.653 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get fsid 2024-03-25T00:58:18.769 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-25T00:58:18.769 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-25T00:58:18.769 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-25T00:58:18.769 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-25T00:58:18.771 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-25T00:58:18.771 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-25T00:58:18.771 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:58:18.771 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:58:18.771 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:58:18.771 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:58:18.771 INFO:tasks.workunit.client.0.smithi080.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:58:18.771 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:58:18.772 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-25T00:58:18.772 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81310/ceph-mon.a.asok config get mon_host 2024-03-25T00:58:18.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-03-25T00:58:18.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-03-25T00:58:18.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-25T00:58:18.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-25T00:58:18.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-25T00:58:18.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-03-25T00:58:18.885 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-25T00:58:19.810 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-25T00:58:19.811 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:58:19.811 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:58:19.811 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:58:19.811 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:58:19.812 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:58:19.812 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:58:19.812 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:58:19.813 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-25T00:58:19.851 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-03-25T00:58:19.852 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-25T00:58:20.210 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'rbd' already exists 2024-03-25T00:58:20.227 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T00:58:21.229 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-03-25T00:58:21.229 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-25T00:58:21.230 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-25T00:58:21.231 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T00:58:21.231 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:58:21.231 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:58:21.231 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-mon.a.asok 2024-03-25T00:58:21.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-03-25T00:58:21.231 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-mon.a.asok log flush 2024-03-25T00:58:21.331 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T00:58:21.343 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-03-25T00:58:21.346 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:58:18.461+0000 7f54056eec80 0 load: jerasure load: lrc 2024-03-25T00:58:21.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2024-03-25T00:58:21.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:316: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2024-03-25T00:58:21.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:317: TEST_rados_get_bad_size_shard_1: setup_osds 4 2024-03-25T00:58:21.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-03-25T00:58:21.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-03-25T00:58:21.348 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-03-25T00:58:21.349 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:58:21.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-03-25T00:58:21.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-03-25T00:58:21.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:58:21.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:58:21.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:58:21.353 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:58:21.353 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:58:21.353 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:58:21.353 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:58:21.353 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:58:21.353 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:58:21.354 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:58:21.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:58:21.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:58:21.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:58:21.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:58:21.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:58:21.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:58:21.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:58:21.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:58:21.356 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:58:21.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:58:21.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:58:21.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:58:21.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-03-25T00:58:21.357 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:58:21.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8fed45a6-1336-4200-9d74-9522422b98d5 2024-03-25T00:58:21.357 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8fed45a6-1336-4200-9d74-9522422b98d5' 2024-03-25T00:58:21.357 INFO:tasks.workunit.client.0.smithi080.stdout:add osd0 8fed45a6-1336-4200-9d74-9522422b98d5 2024-03-25T00:58:21.357 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:58:21.374 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtzABmydLIFhAAGo7xa+hoiZVwvNVbr4vJ8g== 2024-03-25T00:58:21.374 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtzABmydLIFhAAGo7xa+hoiZVwvNVbr4vJ8g=="}' 2024-03-25T00:58:21.374 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8fed45a6-1336-4200-9d74-9522422b98d5 -i td/test-erasure-eio/0/new.json 2024-03-25T00:58:21.736 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:58:21.752 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-03-25T00:58:21.754 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtzABmydLIFhAAGo7xa+hoiZVwvNVbr4vJ8g== --osd-uuid 8fed45a6-1336-4200-9d74-9522422b98d5 2024-03-25T00:58:21.785 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:21.789+0000 7fb41e18e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:58:21.821 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:21.825+0000 7fb41e18e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:58:21.823 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:21.829+0000 7fb41e18e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:58:21.824 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:21.829+0000 7fb41e18e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:58:21.824 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:21.829+0000 7fb41e18e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:58:21.824 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:21.829+0000 7fb41e18e7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-03-25T00:58:21.861 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:21.865+0000 7fb41e18e7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-03-25T00:58:29.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-03-25T00:58:29.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:58:29.667 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd0 key to auth repository 2024-03-25T00:58:29.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-25T00:58:29.667 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:58:30.229 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.0 2024-03-25T00:58:30.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-25T00:58:30.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:58:30.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:58:30.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:58:30.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:58:30.261 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:30.265+0000 7f5c6f6477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:58:30.263 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:30.269+0000 7f5c6f6477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:58:30.264 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:30.269+0000 7f5c6f6477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:58:30.618 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-25T00:58:30.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:58:30.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-25T00:58:30.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:58:30.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:58:30.619 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:30.619 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:58:30.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:58:30.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:30.620 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:30.802 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:30.805+0000 7f5c6f6477c0 -1 Falling back to public interface 2024-03-25T00:58:30.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:31.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:31.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:31.997 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:58:31.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:58:31.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:31.997 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:32.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:33.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:33.408 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:33.408 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:58:33.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:58:33.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:33.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:33.721 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:58:33.725+0000 7f5c6f6477c0 -1 osd.0 0 log_to_monitors true 2024-03-25T00:58:33.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:34.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:34.863 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:34.863 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:58:34.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:58:34.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:34.864 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:35.497 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:36.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:36.499 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:36.499 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:58:36.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:58:36.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:36.500 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:37.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:38.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:38.046 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:38.046 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:58:38.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:58:38.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:38.047 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:38.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:39.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:39.529 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:39.529 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:58:39.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:58:39.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:39.530 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:40.124 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:41.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:41.126 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:41.126 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:58:41.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:58:41.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:41.127 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:41.555 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:42.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:42.557 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:42.557 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:58:42.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:58:42.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:42.558 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:43.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:44.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:44.162 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:44.162 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:58:44.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:58:44.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:44.163 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:44.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:45.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:45.683 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:45.683 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:58:45.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:58:45.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:45.684 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:46.228 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:47.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:47.229 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:47.230 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:58:47.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:58:47.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:47.230 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:47.908 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:48.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:48.910 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:48.911 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:58:48.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:58:48.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:48.911 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:49.551 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:50.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:50.552 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:50.553 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:58:50.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:58:50.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:50.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:51.078 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:52.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:52.079 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:52.080 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:58:52.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:58:52.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:52.080 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:52.575 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:53.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:53.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:53.577 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:58:53.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:58:53.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:53.577 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:54.404 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:55.406 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:55.406 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:58:55.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:55.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:58:55.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:55.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:56.264 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:57.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:57.266 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:57.267 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:58:57.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:58:57.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:57.267 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:58:58.623 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:58:59.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:58:59.624 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:58:59.625 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:58:59.625 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:58:59.625 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:58:59.625 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:59:00.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:01.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:01.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:01.526 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-25T00:59:01.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T00:59:01.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:01.527 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:59:02.255 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:03.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:03.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:03.257 INFO:tasks.workunit.client.0.smithi080.stdout:20 2024-03-25T00:59:03.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T00:59:03.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:03.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:59:04.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:05.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:05.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:05.159 INFO:tasks.workunit.client.0.smithi080.stdout:21 2024-03-25T00:59:05.159 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-25T00:59:05.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:05.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-25T00:59:05.779 INFO:tasks.workunit.client.0.smithi080.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1636478105,v1:127.0.0.1:6803/1636478105] [v2:127.0.0.1:6804/1636478105,v1:127.0.0.1:6805/1636478105] exists,up 8fed45a6-1336-4200-9d74-9522422b98d5 2024-03-25T00:59:05.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:59:05.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:59:05.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:59:05.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:59:05.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-03-25T00:59:05.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:59:05.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:59:05.783 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:59:05.784 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:59:05.785 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:59:05.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:59:05.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:59:05.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:59:05.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:59:05.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:59:05.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:59:05.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T00:59:05.787 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:59:05.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=60b82a00-8e41-4adb-8e81-6f9da081f0b1 2024-03-25T00:59:05.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 60b82a00-8e41-4adb-8e81-6f9da081f0b1' 2024-03-25T00:59:05.787 INFO:tasks.workunit.client.0.smithi080.stdout:add osd1 60b82a00-8e41-4adb-8e81-6f9da081f0b1 2024-03-25T00:59:05.787 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:59:05.807 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZzABmUTeuMBAAJJJky2f5XINc2PDR3jynfw== 2024-03-25T00:59:05.807 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZzABmUTeuMBAAJJJky2f5XINc2PDR3jynfw=="}' 2024-03-25T00:59:05.807 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 60b82a00-8e41-4adb-8e81-6f9da081f0b1 -i td/test-erasure-eio/1/new.json 2024-03-25T00:59:06.562 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:59:06.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-03-25T00:59:06.581 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZzABmUTeuMBAAJJJky2f5XINc2PDR3jynfw== --osd-uuid 60b82a00-8e41-4adb-8e81-6f9da081f0b1 2024-03-25T00:59:06.615 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:06.624+0000 7fad6e7ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:59:06.617 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:06.624+0000 7fad6e7ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:59:06.619 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:06.628+0000 7fad6e7ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:59:06.619 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:06.628+0000 7fad6e7ea7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:59:06.619 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:06.628+0000 7fad6e7ea7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:59:06.620 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:06.628+0000 7fad6e7ea7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-03-25T00:59:06.679 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:06.688+0000 7fad6e7ea7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-03-25T00:59:14.686 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-03-25T00:59:14.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T00:59:14.688 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd1 key to auth repository 2024-03-25T00:59:14.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-25T00:59:14.688 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T00:59:15.225 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.1 2024-03-25T00:59:15.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-25T00:59:15.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T00:59:15.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T00:59:15.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T00:59:15.226 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T00:59:15.258 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:15.268+0000 7f241fcb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:59:15.260 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:15.268+0000 7f241fcb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:59:15.261 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:15.268+0000 7f241fcb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:59:15.638 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-25T00:59:15.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T00:59:15.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T00:59:15.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T00:59:15.639 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T00:59:15.639 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T00:59:15.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:15.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T00:59:15.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:15.640 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:15.803 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:15.812+0000 7f241fcb27c0 -1 Falling back to public interface 2024-03-25T00:59:16.028 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:17.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:17.030 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:17.031 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T00:59:17.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T00:59:17.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:17.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:17.453 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:18.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:18.455 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:18.455 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:59:18.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T00:59:18.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:18.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:18.673 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:18.680+0000 7f241fcb27c0 -1 osd.1 0 log_to_monitors true 2024-03-25T00:59:18.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:19.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:19.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:19.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T00:59:19.996 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T00:59:19.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:19.996 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:20.536 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:21.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:21.537 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:21.538 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T00:59:21.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T00:59:21.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:21.538 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:21.998 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:22.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:22.999 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:23.000 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T00:59:23.000 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T00:59:23.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:23.001 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:23.604 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:24.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:24.605 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:24.606 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T00:59:24.606 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T00:59:24.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:24.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:25.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:26.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:26.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:26.206 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T00:59:26.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T00:59:26.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:26.207 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:26.952 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:27.954 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:27.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:27.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T00:59:27.955 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T00:59:27.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:27.956 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:28.464 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:29.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:29.466 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:29.467 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T00:59:29.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T00:59:29.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:29.467 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:30.087 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:31.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:31.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:31.091 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T00:59:31.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T00:59:31.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:31.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:31.582 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:32.584 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:32.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:32.585 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T00:59:32.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T00:59:32.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:32.586 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:33.234 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:34.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:34.235 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:34.236 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T00:59:34.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T00:59:34.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:34.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:34.868 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:35.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:35.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:35.871 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T00:59:35.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T00:59:35.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:35.871 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:36.340 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:37.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:37.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:37.342 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T00:59:37.342 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T00:59:37.343 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:37.343 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:37.890 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:38.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:38.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:38.892 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T00:59:38.892 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T00:59:38.893 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:38.893 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:40.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:41.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:41.042 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:41.043 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T00:59:41.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T00:59:41.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:41.044 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:42.004 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:43.005 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:43.006 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:43.006 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T00:59:43.006 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T00:59:43.006 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:43.007 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:43.992 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:44.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:44.994 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:44.994 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T00:59:44.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T00:59:44.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:44.995 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:46.021 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:47.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:47.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:47.023 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T00:59:47.023 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-25T00:59:47.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:47.024 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:47.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:48.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:48.689 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:48.690 INFO:tasks.workunit.client.0.smithi080.stdout:20 2024-03-25T00:59:48.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T00:59:48.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:48.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:49.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T00:59:50.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T00:59:50.946 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T00:59:50.946 INFO:tasks.workunit.client.0.smithi080.stdout:21 2024-03-25T00:59:50.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-25T00:59:50.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T00:59:50.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T00:59:51.700 INFO:tasks.workunit.client.0.smithi080.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1548649047,v1:127.0.0.1:6811/1548649047] [v2:127.0.0.1:6812/1548649047,v1:127.0.0.1:6813/1548649047] exists,up 60b82a00-8e41-4adb-8e81-6f9da081f0b1 2024-03-25T00:59:51.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T00:59:51.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T00:59:51.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T00:59:51.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T00:59:51.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-03-25T00:59:51.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T00:59:51.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T00:59:51.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-25T00:59:51.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T00:59:51.704 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T00:59:51.705 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T00:59:51.706 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T00:59:51.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T00:59:51.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T00:59:51.707 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T00:59:51.707 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ae1444d-8ac1-4a9a-8795-e9ce74611232 2024-03-25T00:59:51.707 INFO:tasks.workunit.client.0.smithi080.stdout:add osd2 7ae1444d-8ac1-4a9a-8795-e9ce74611232 2024-03-25T00:59:51.708 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7ae1444d-8ac1-4a9a-8795-e9ce74611232' 2024-03-25T00:59:51.708 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T00:59:51.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHzABmW3MTLBAAcJuHZC3lurPoEcfewjARpA== 2024-03-25T00:59:51.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHzABmW3MTLBAAcJuHZC3lurPoEcfewjARpA=="}' 2024-03-25T00:59:51.729 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ae1444d-8ac1-4a9a-8795-e9ce74611232 -i td/test-erasure-eio/2/new.json 2024-03-25T00:59:52.708 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T00:59:52.726 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-03-25T00:59:52.728 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHzABmW3MTLBAAcJuHZC3lurPoEcfewjARpA== --osd-uuid 7ae1444d-8ac1-4a9a-8795-e9ce74611232 2024-03-25T00:59:52.765 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:52.775+0000 7fa61284f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:59:52.768 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:52.775+0000 7fa61284f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:59:52.769 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:52.779+0000 7fa61284f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T00:59:52.770 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:52.779+0000 7fa61284f7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:59:52.770 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:52.779+0000 7fa61284f7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:59:52.770 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:52.779+0000 7fa61284f7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-03-25T00:59:52.942 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T00:59:52.951+0000 7fa61284f7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-03-25T01:00:00.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-03-25T01:00:00.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T01:00:00.743 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd2 key to auth repository 2024-03-25T01:00:00.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-25T01:00:00.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T01:00:01.320 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.2 2024-03-25T01:00:01.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-25T01:00:01.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T01:00:01.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T01:00:01.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T01:00:01.321 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T01:00:01.346 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:01.355+0000 7f658a9ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:00:01.348 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:01.355+0000 7f658a9ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:00:01.350 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:01.359+0000 7f658a9ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:00:01.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-25T01:00:01.736 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T01:00:01.736 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T01:00:01.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T01:00:01.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T01:00:01.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T01:00:01.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:01.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T01:00:01.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:01.737 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:01.898 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:01.907+0000 7f658a9ae7c0 -1 Falling back to public interface 2024-03-25T01:00:02.156 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:03.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:03.157 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:03.158 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T01:00:03.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T01:00:03.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:03.158 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:03.563 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:04.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:04.565 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:04.565 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T01:00:04.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T01:00:04.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:04.566 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:04.752 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:04.759+0000 7f658a9ae7c0 -1 osd.2 0 log_to_monitors true 2024-03-25T01:00:05.074 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:06.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:06.076 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:06.076 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T01:00:06.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T01:00:06.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:06.077 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:06.544 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:07.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:07.546 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:07.546 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T01:00:07.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T01:00:07.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:07.547 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:08.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:09.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:09.257 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:09.258 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T01:00:09.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T01:00:09.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:09.258 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:09.793 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:10.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:10.796 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:10.796 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T01:00:10.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T01:00:10.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:10.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:11.394 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:12.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:12.396 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:12.396 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T01:00:12.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T01:00:12.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:12.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:13.035 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:14.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:14.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:14.038 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T01:00:14.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T01:00:14.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:14.038 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:14.518 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:15.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:15.520 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:15.521 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T01:00:15.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T01:00:15.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:15.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:16.212 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:17.213 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:17.214 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:17.214 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T01:00:17.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T01:00:17.215 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:17.216 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:17.842 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:18.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:18.844 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:18.845 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T01:00:18.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T01:00:18.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:18.845 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:19.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:20.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:20.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:20.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T01:00:20.488 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T01:00:20.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:20.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:21.051 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:22.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:22.053 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:22.054 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T01:00:22.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T01:00:22.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:22.054 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:22.485 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:23.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:23.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:23.487 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T01:00:23.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T01:00:23.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:23.488 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:24.193 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:25.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:25.195 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:25.195 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T01:00:25.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T01:00:25.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:25.196 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:26.444 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:27.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:27.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:27.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T01:00:27.447 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T01:00:27.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:27.447 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:28.297 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:29.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:29.298 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:29.299 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T01:00:29.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T01:00:29.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:29.299 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:30.152 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:31.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:31.154 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:31.155 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T01:00:31.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T01:00:31.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:31.155 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:32.320 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:33.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:33.322 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:33.322 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-25T01:00:33.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T01:00:33.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:33.323 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:34.234 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:35.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:35.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:35.236 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T01:00:35.237 INFO:tasks.workunit.client.0.smithi080.stdout:20 2024-03-25T01:00:35.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:35.237 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:36.062 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:37.063 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:37.064 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:37.064 INFO:tasks.workunit.client.0.smithi080.stdout:21 2024-03-25T01:00:37.064 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-25T01:00:37.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:37.065 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:00:37.750 INFO:tasks.workunit.client.0.smithi080.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4177759779,v1:127.0.0.1:6819/4177759779] [v2:127.0.0.1:6820/4177759779,v1:127.0.0.1:6821/4177759779] exists,up 7ae1444d-8ac1-4a9a-8795-e9ce74611232 2024-03-25T01:00:37.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T01:00:37.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T01:00:37.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T01:00:37.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-03-25T01:00:37.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-03-25T01:00:37.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-03-25T01:00:37.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-25T01:00:37.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-25T01:00:37.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-25T01:00:37.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-03-25T01:00:37.756 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T01:00:37.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T01:00:37.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T01:00:37.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T01:00:37.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-03-25T01:00:37.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-03-25T01:00:37.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-25T01:00:37.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-25T01:00:37.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T01:00:37.757 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-25T01:00:37.758 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T01:00:37.758 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T01:00:37.758 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T01:00:37.758 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T01:00:37.759 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T01:00:37.759 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T01:00:37.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T01:00:37.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-25T01:00:37.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-25T01:00:37.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-25T01:00:37.759 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T01:00:37.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T01:00:37.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T01:00:37.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T01:00:37.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T01:00:37.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T01:00:37.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-25T01:00:37.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-25T01:00:37.760 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-03-25T01:00:37.761 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-25T01:00:37.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=80dcd9d5-791a-4e7d-a752-12fbcd2fc846 2024-03-25T01:00:37.761 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 80dcd9d5-791a-4e7d-a752-12fbcd2fc846' 2024-03-25T01:00:37.761 INFO:tasks.workunit.client.0.smithi080.stdout:add osd3 80dcd9d5-791a-4e7d-a752-12fbcd2fc846 2024-03-25T01:00:37.761 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-25T01:00:37.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1zABmxDY3LxAAD47oUbPIr/ucwc/PAtSa7Q== 2024-03-25T01:00:37.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1zABmxDY3LxAAD47oUbPIr/ucwc/PAtSa7Q=="}' 2024-03-25T01:00:37.781 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 80dcd9d5-791a-4e7d-a752-12fbcd2fc846 -i td/test-erasure-eio/3/new.json 2024-03-25T01:00:39.198 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T01:00:39.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-03-25T01:00:39.221 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1zABmxDY3LxAAD47oUbPIr/ucwc/PAtSa7Q== --osd-uuid 80dcd9d5-791a-4e7d-a752-12fbcd2fc846 2024-03-25T01:00:39.255 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:39.262+0000 7f7339a1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:00:39.264 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:39.274+0000 7f7339a1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:00:39.265 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:39.274+0000 7f7339a1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:00:39.266 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:39.274+0000 7f7339a1d7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T01:00:39.266 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:39.274+0000 7f7339a1d7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T01:00:39.266 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:39.274+0000 7f7339a1d7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-03-25T01:00:39.365 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:39.374+0000 7f7339a1d7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-03-25T01:00:47.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-03-25T01:00:47.372 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-25T01:00:47.373 INFO:tasks.workunit.client.0.smithi080.stdout:adding osd3 key to auth repository 2024-03-25T01:00:47.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-25T01:00:47.373 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-25T01:00:47.954 INFO:tasks.workunit.client.0.smithi080.stdout:start osd.3 2024-03-25T01:00:47.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-25T01:00:47.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-25T01:00:47.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T01:00:47.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-25T01:00:47.955 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-25T01:00:47.985 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:47.994+0000 7fd80b9557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:00:47.988 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:47.994+0000 7fd80b9557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:00:47.989 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:47.998+0000 7fd80b9557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:00:48.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-25T01:00:48.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T01:00:48.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-25T01:00:48.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T01:00:48.352 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T01:00:48.352 INFO:tasks.workunit.client.0.smithi080.stdout:0 2024-03-25T01:00:48.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:48.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T01:00:48.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:48.353 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:00:48.521 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:48.530+0000 7fd80b9557c0 -1 Falling back to public interface 2024-03-25T01:00:48.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:49.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:49.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:49.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T01:00:49.788 INFO:tasks.workunit.client.0.smithi080.stdout:1 2024-03-25T01:00:49.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:49.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:00:50.204 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:51.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:51.205 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:51.206 INFO:tasks.workunit.client.0.smithi080.stdout:2 2024-03-25T01:00:51.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T01:00:51.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:51.206 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:00:51.380 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:00:51.390+0000 7fd80b9557c0 -1 osd.3 0 log_to_monitors true 2024-03-25T01:00:51.687 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:52.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:52.690 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:52.690 INFO:tasks.workunit.client.0.smithi080.stdout:3 2024-03-25T01:00:52.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T01:00:52.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:52.691 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:00:53.386 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:54.387 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:54.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:54.388 INFO:tasks.workunit.client.0.smithi080.stdout:4 2024-03-25T01:00:54.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-25T01:00:54.388 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:54.389 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:00:55.031 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:56.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:56.033 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:56.033 INFO:tasks.workunit.client.0.smithi080.stdout:5 2024-03-25T01:00:56.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-25T01:00:56.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:56.034 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:00:56.576 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:57.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:57.578 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:57.578 INFO:tasks.workunit.client.0.smithi080.stdout:6 2024-03-25T01:00:57.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-25T01:00:57.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:57.579 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:00:58.188 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:00:59.190 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:00:59.190 INFO:tasks.workunit.client.0.smithi080.stdout:7 2024-03-25T01:00:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:00:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-25T01:00:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:00:59.191 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:00:59.884 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:00.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:00.886 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:00.886 INFO:tasks.workunit.client.0.smithi080.stdout:8 2024-03-25T01:01:00.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-25T01:01:00.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:00.887 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:01.469 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:02.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:02.471 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:02.471 INFO:tasks.workunit.client.0.smithi080.stdout:9 2024-03-25T01:01:02.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-25T01:01:02.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:02.472 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:02.947 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:03.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:03.949 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:03.949 INFO:tasks.workunit.client.0.smithi080.stdout:10 2024-03-25T01:01:03.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-25T01:01:03.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:03.950 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:04.487 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:05.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:05.489 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:05.489 INFO:tasks.workunit.client.0.smithi080.stdout:11 2024-03-25T01:01:05.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-25T01:01:05.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:05.490 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:06.101 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:07.103 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:07.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:07.104 INFO:tasks.workunit.client.0.smithi080.stdout:12 2024-03-25T01:01:07.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-25T01:01:07.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:07.105 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:07.766 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:08.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:08.768 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:08.769 INFO:tasks.workunit.client.0.smithi080.stdout:13 2024-03-25T01:01:08.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-25T01:01:08.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:08.769 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:09.324 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:10.325 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:10.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:10.326 INFO:tasks.workunit.client.0.smithi080.stdout:14 2024-03-25T01:01:10.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-25T01:01:10.326 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:10.327 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:10.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:11.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:11.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:11.788 INFO:tasks.workunit.client.0.smithi080.stdout:15 2024-03-25T01:01:11.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-25T01:01:11.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:11.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:12.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:13.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:13.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:13.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-25T01:01:13.940 INFO:tasks.workunit.client.0.smithi080.stdout:16 2024-03-25T01:01:13.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:13.940 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:14.795 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:15.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:15.797 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:15.798 INFO:tasks.workunit.client.0.smithi080.stdout:17 2024-03-25T01:01:15.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-25T01:01:15.798 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:15.799 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:16.345 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:17.346 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:17.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:17.347 INFO:tasks.workunit.client.0.smithi080.stdout:18 2024-03-25T01:01:17.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-25T01:01:17.347 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:17.348 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:18.874 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:19.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:19.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:19.877 INFO:tasks.workunit.client.0.smithi080.stdout:19 2024-03-25T01:01:19.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-25T01:01:19.877 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:19.878 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:20.674 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:21.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:21.676 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:21.677 INFO:tasks.workunit.client.0.smithi080.stdout:20 2024-03-25T01:01:21.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-03-25T01:01:21.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:21.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:22.430 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:23.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:23.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:23.432 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-03-25T01:01:23.432 INFO:tasks.workunit.client.0.smithi080.stdout:21 2024-03-25T01:01:23.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:23.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:24.037 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:25.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:25.039 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:25.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-03-25T01:01:25.040 INFO:tasks.workunit.client.0.smithi080.stdout:22 2024-03-25T01:01:25.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:25.040 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-25T01:01:25.662 INFO:tasks.workunit.client.0.smithi080.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2656762557,v1:127.0.0.1:6827/2656762557] [v2:127.0.0.1:6828/2656762557,v1:127.0.0.1:6829/2656762557] exists,up 80dcd9d5-791a-4e7d-a752-12fbcd2fc846 2024-03-25T01:01:25.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T01:01:25.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T01:01:25.663 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T01:01:25.664 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-03-25T01:01:25.664 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-25T01:01:25.664 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-25T01:01:25.664 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-25T01:01:25.664 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T01:01:25.664 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T01:01:25.664 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81310/ceph-osd.0.asok 2024-03-25T01:01:25.664 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-03-25T01:01:25.665 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81310/ceph-osd.0.asok log flush 2024-03-25T01:01:25.762 INFO:tasks.workunit.client.0.smithi080.stdout:{} 2024-03-25T01:01:25.773 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-03-25T01:01:25.778 INFO:tasks.workunit.client.0.smithi080.stdout:2024-03-25T00:58:30.813+0000 7f5c6f6477c0 0 load: jerasure load: lrc 2024-03-25T01:01:25.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:319: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2024-03-25T01:01:25.779 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:320: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-03-25T01:01:25.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-03-25T01:01:25.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-03-25T01:01:25.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-03-25T01:01:25.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-03-25T01:01:25.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-03-25T01:01:25.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-03-25T01:01:25.780 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-03-25T01:01:27.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-03-25T01:01:27.096 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-03-25T01:01:28.649 INFO:tasks.workunit.client.0.smithi080.stderr:pool 'pool-jerasure' already exists 2024-03-25T01:01:28.668 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-25T01:01:29.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-03-25T01:01:29.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T01:01:29.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T01:01:29.671 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T01:01:29.671 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T01:01:29.672 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T01:01:29.672 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T01:01:29.673 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T01:01:29.673 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T01:01:29.673 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T01:01:29.803 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T01:01:29.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T01:01:29.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T01:01:29.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T01:01:29.804 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T01:01:29.804 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T01:01:30.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T01:01:30.220 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T01:01:30.220 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T01:01:30.220 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T01:01:30.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T01:01:30.220 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:01:30.221 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T01:01:30.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803814 2024-03-25T01:01:30.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803814 2024-03-25T01:01:30.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803814' 2024-03-25T01:01:30.360 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:01:30.360 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T01:01:30.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574876 2024-03-25T01:01:30.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574876 2024-03-25T01:01:30.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803814 1-55834574876' 2024-03-25T01:01:30.501 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:01:30.501 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T01:01:30.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378644 2024-03-25T01:01:30.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378644 2024-03-25T01:01:30.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803814 1-55834574876 2-81604378644' 2024-03-25T01:01:30.696 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:01:30.696 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T01:01:30.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084298 2024-03-25T01:01:30.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084298 2024-03-25T01:01:30.857 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803814 1-55834574876 2-81604378644 3-120259084298' 2024-03-25T01:01:30.858 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:01:30.859 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803814 2024-03-25T01:01:30.859 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:01:30.859 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T01:01:30.860 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803814 2024-03-25T01:01:30.860 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:01:30.861 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803814 2024-03-25T01:01:30.861 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.0 seq 25769803814 2024-03-25T01:01:30.862 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803814' 2024-03-25T01:01:30.863 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T01:01:31.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803814 -lt 25769803814 2024-03-25T01:01:31.279 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:01:31.280 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:01:31.280 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574876 2024-03-25T01:01:31.282 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T01:01:31.282 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574876 2024-03-25T01:01:31.282 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:01:31.283 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574876 2024-03-25T01:01:31.284 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.1 seq 55834574876 2024-03-25T01:01:31.284 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574876' 2024-03-25T01:01:31.284 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T01:01:31.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574874 -lt 55834574876 2024-03-25T01:01:31.677 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T01:01:32.678 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T01:01:32.679 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T01:01:33.091 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574877 -lt 55834574876 2024-03-25T01:01:33.092 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:01:33.092 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378644 2024-03-25T01:01:33.092 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:01:33.095 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T01:01:33.096 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378644 2024-03-25T01:01:33.096 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:01:33.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378644 2024-03-25T01:01:33.097 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.2 seq 81604378644 2024-03-25T01:01:33.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378644' 2024-03-25T01:01:33.097 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T01:01:33.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378644 -lt 81604378644 2024-03-25T01:01:33.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:01:33.508 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084298 2024-03-25T01:01:33.509 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:01:33.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T01:01:33.511 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084298 2024-03-25T01:01:33.511 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:01:33.512 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084298 2024-03-25T01:01:33.513 INFO:tasks.workunit.client.0.smithi080.stdout:waiting osd.3 seq 120259084298 2024-03-25T01:01:33.513 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084298' 2024-03-25T01:01:33.513 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T01:01:33.906 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084298 -lt 120259084298 2024-03-25T01:01:33.906 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T01:01:33.907 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T01:01:33.907 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T01:01:34.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T01:01:34.456 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T01:01:34.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T01:01:34.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T01:01:34.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T01:01:34.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T01:01:34.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T01:01:34.457 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T01:01:34.899 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T01:01:34.900 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T01:01:34.900 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T01:01:34.900 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T01:01:35.504 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T01:01:35.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T01:01:35.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T01:01:35.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_1: local shard_id=1 2024-03-25T01:01:35.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2024-03-25T01:01:35.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-03-25T01:01:35.505 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-03-25T01:01:35.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-03-25T01:01:35.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-03-25T01:01:35.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-03-25T01:01:35.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-03-25T01:01:35.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-03-25T01:01:35.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-03-25T01:01:35.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-81310-1-10 2024-03-25T01:01:35.506 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81310-1-10 2024-03-25T01:01:35.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-03-25T01:01:35.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-03-25T01:01:35.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81310-1-10 2024-03-25T01:01:35.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T01:01:35.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-03-25T01:01:35.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T01:01:35.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-03-25T01:01:35.507 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T01:01:35.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-03-25T01:01:35.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-03-25T01:01:35.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-03-25T01:01:35.508 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81310-1-10 td/test-erasure-eio/ORIGINAL 2024-03-25T01:01:35.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-81310-1-10 td/test-erasure-eio 1 10 set 2024-03-25T01:01:35.720 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81310-1-10 2024-03-25T01:01:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-03-25T01:01:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-03-25T01:01:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-03-25T01:01:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-03-25T01:01:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-03-25T01:01:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-03-25T01:01:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-03-25T01:01:35.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-03-25T01:01:35.722 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-03-25T01:01:35.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81310-1-10 2024-03-25T01:01:35.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T01:01:35.722 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81310-1-10 2024-03-25T01:01:35.723 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81310-1-10 2024-03-25T01:01:35.723 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T01:01:36.149 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T01:01:36.149 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T01:01:36.149 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-25T01:01:36.149 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-03-25T01:01:36.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-03-25T01:01:36.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-03-25T01:01:36.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-03-25T01:01:36.149 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-03-25T01:01:36.961 INFO:tasks.workunit.client.0.smithi080.stderr:noout is set 2024-03-25T01:01:36.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-03-25T01:01:36.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-03-25T01:01:36.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-03-25T01:01:36.986 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records in 2024-03-25T01:01:36.986 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records out 2024-03-25T01:01:36.986 INFO:tasks.workunit.client.0.smithi080.stderr:10 bytes copied, 0.000121254 s, 82.5 kB/s 2024-03-25T01:01:36.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81310-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T01:01:36.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-03-25T01:01:36.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T01:01:36.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-25T01:01:36.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T01:01:36.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81310-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T01:01:36.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-03-25T01:01:36.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T01:01:36.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-25T01:01:36.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T01:01:36.987 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-03-25T01:01:36.988 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T01:01:36.988 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T01:01:36.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T01:01:36.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T01:01:36.988 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T01:01:38.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T01:01:38.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81310-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T01:01:38.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-03-25T01:01:38.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T01:01:38.300 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-25T01:01:38.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T01:01:38.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-03-25T01:01:38.301 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81310-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T01:01:40.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-03-25T01:01:40.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T01:01:40.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T01:01:40.912 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-25T01:01:40.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T01:01:40.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-03-25T01:01:40.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T01:01:40.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T01:01:40.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T01:01:40.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T01:01:40.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-03-25T01:01:40.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-03-25T01:01:40.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T01:01:40.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T01:01:40.913 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T01:01:40.914 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T01:01:40.914 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T01:01:40.914 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T01:01:40.914 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T01:01:40.914 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T01:01:40.915 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T01:01:40.915 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T01:01:40.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T01:01:40.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T01:01:40.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T01:01:40.915 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T01:01:40.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T01:01:40.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T01:01:40.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T01:01:40.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T01:01:40.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T01:01:40.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T01:01:40.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-03-25T01:01:40.916 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-25T01:01:40.916 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.1 2024-03-25T01:01:40.917 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T01:01:40.918 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-03-25T01:01:40.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-25T01:01:40.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T01:01:40.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T01:01:40.919 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T01:01:40.951 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:01:40.961+0000 7fbb760427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:01:40.979 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:01:40.989+0000 7fbb760427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:01:40.982 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:01:40.989+0000 7fbb760427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:01:41.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-25T01:01:41.363 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T01:01:41.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-25T01:01:41.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T01:01:41.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T01:01:41.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:41.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T01:01:41.364 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T01:01:41.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:41.364 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T01:01:41.536 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:01:41.545+0000 7fbb760427c0 -1 Falling back to public interface 2024-03-25T01:01:41.786 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:42.787 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:42.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:42.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T01:01:42.788 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T01:01:42.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:42.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T01:01:43.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:44.238 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:44.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:44.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T01:01:44.239 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T01:01:44.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:44.239 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T01:01:44.455 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:01:44.465+0000 7fbb760427c0 -1 osd.1 38 log_to_monitors true 2024-03-25T01:01:44.679 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:01:45.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:01:45.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:01:45.681 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T01:01:45.681 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T01:01:45.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:01:45.682 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-25T01:01:46.397 INFO:tasks.workunit.client.0.smithi080.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 40 last_clean_interval [13,38) [v2:127.0.0.1:6810/2203703283,v1:127.0.0.1:6811/2203703283] [v2:127.0.0.1:6812/2203703283,v1:127.0.0.1:6813/2203703283] exists,up 60b82a00-8e41-4adb-8e81-6f9da081f0b1 2024-03-25T01:01:46.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T01:01:46.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T01:01:46.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T01:01:46.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T01:01:46.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T01:01:46.397 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T01:01:46.398 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T01:01:46.399 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T01:01:46.399 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T01:01:46.399 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T01:01:46.399 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T01:01:46.399 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T01:01:46.399 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T01:01:46.509 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T01:01:46.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T01:01:46.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T01:01:46.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T01:01:46.510 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T01:01:46.510 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T01:01:47.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T01:01:47.097 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T01:01:47.097 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T01:01:47.097 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T01:01:47.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T01:01:47.097 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:01:47.098 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T01:01:47.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803820 2024-03-25T01:01:47.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803820 2024-03-25T01:01:47.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820' 2024-03-25T01:01:47.293 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:01:47.293 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T01:01:47.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561027 2024-03-25T01:01:47.445 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561027 2024-03-25T01:01:47.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820 1-188978561027' 2024-03-25T01:01:47.446 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:01:47.446 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T01:01:47.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378650 2024-03-25T01:01:47.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378650 2024-03-25T01:01:47.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820 1-188978561027 2-81604378650' 2024-03-25T01:01:47.589 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:01:47.589 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T01:01:47.742 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084303 2024-03-25T01:01:47.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084303 2024-03-25T01:01:47.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803820 1-188978561027 2-81604378650 3-120259084303' 2024-03-25T01:01:47.743 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:01:47.743 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803820 2024-03-25T01:01:47.743 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:01:47.744 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T01:01:47.744 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803820 2024-03-25T01:01:47.744 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:01:47.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803820 2024-03-25T01:01:47.746 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803820' 2024-03-25T01:01:47.746 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 25769803820 2024-03-25T01:01:47.746 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T01:01:48.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803818 -lt 25769803820 2024-03-25T01:01:48.181 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-25T01:01:49.183 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-25T01:01:49.184 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T01:01:49.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803820 -lt 25769803820 2024-03-25T01:01:49.583 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:01:49.584 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:01:49.584 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561027 2024-03-25T01:01:49.585 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T01:01:49.586 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561027 2024-03-25T01:01:49.586 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:01:49.587 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561027 2024-03-25T01:01:49.588 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-03-25T01:01:49.588 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 188978561027 2024-03-25T01:01:49.588 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T01:01:49.981 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561027 -lt 188978561027 2024-03-25T01:01:49.982 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:01:49.982 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378650 2024-03-25T01:01:49.982 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:01:49.983 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T01:01:49.984 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378650 2024-03-25T01:01:49.984 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:01:49.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378650 2024-03-25T01:01:49.985 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378650' 2024-03-25T01:01:49.986 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 81604378650 2024-03-25T01:01:49.986 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T01:01:50.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378650 -lt 81604378650 2024-03-25T01:01:50.405 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:01:50.406 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084303 2024-03-25T01:01:50.406 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:01:50.407 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T01:01:50.408 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084303 2024-03-25T01:01:50.408 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:01:50.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084303 2024-03-25T01:01:50.409 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084303' 2024-03-25T01:01:50.409 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 120259084303 2024-03-25T01:01:50.409 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T01:01:50.843 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084304 -lt 120259084303 2024-03-25T01:01:50.844 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T01:01:50.845 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T01:01:50.845 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T01:01:51.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T01:01:51.433 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T01:01:51.434 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T01:01:51.434 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T01:01:51.434 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T01:01:51.435 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T01:01:51.435 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T01:01:51.435 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T01:01:51.837 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-25T01:01:51.838 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T01:01:51.838 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T01:01:51.838 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T01:01:52.424 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-25T01:01:52.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-25T01:01:52.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-25T01:01:52.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-25T01:01:52.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-25T01:01:52.425 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-25T01:01:52.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-25T01:01:52.526 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T01:01:52.527 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T01:01:52.527 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T01:01:52.527 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T01:01:52.527 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T01:01:52.527 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T01:01:52.528 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T01:01:52.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T01:01:52.936 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T01:01:52.937 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T01:01:52.937 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T01:01:53.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T01:01:53.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T01:01:53.493 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T01:01:53.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-03-25T01:01:53.494 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-03-25T01:01:54.176 INFO:tasks.workunit.client.0.smithi080.stderr:noout is unset 2024-03-25T01:01:54.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81310-1-10 2024-03-25T01:01:54.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T01:01:54.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T01:01:54.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81310-1-10 2024-03-25T01:01:54.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-03-25T01:01:54.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-03-25T01:01:54.197 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81310-1-10 td/test-erasure-eio/COPY 2024-03-25T01:01:54.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-03-25T01:01:54.416 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-03-25T01:01:54.418 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-03-25T01:01:54.420 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-03-25T01:01:54.420 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-81310-1-10 td/test-erasure-eio 2 10 set 2024-03-25T01:01:54.420 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81310-1-10 2024-03-25T01:01:54.420 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-03-25T01:01:54.420 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-03-25T01:01:54.420 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-03-25T01:01:54.420 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-03-25T01:01:54.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-03-25T01:01:54.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-03-25T01:01:54.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-03-25T01:01:54.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-03-25T01:01:54.421 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-03-25T01:01:54.421 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81310-1-10 2024-03-25T01:01:54.421 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-03-25T01:01:54.421 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81310-1-10 2024-03-25T01:01:54.422 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81310-1-10 2024-03-25T01:01:54.422 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-25T01:01:54.869 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-25T01:01:54.869 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T01:01:54.869 INFO:tasks.workunit.client.0.smithi080.stderr:2' 2024-03-25T01:01:54.869 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-03-25T01:01:54.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-03-25T01:01:54.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-03-25T01:01:54.869 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-03-25T01:01:54.870 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-03-25T01:01:55.766 INFO:tasks.workunit.client.0.smithi080.stderr:noout is set 2024-03-25T01:01:55.788 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-03-25T01:01:55.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-03-25T01:01:55.789 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-03-25T01:01:55.790 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records in 2024-03-25T01:01:55.790 INFO:tasks.workunit.client.0.smithi080.stderr:1+0 records out 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:10 bytes copied, 0.000121615 s, 82.2 kB/s 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-81310-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-81310-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-25T01:01:55.791 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-03-25T01:01:55.792 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T01:01:55.792 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T01:01:55.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T01:01:55.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T01:01:55.792 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T01:01:57.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T01:01:57.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-81310-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T01:01:57.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-03-25T01:01:57.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-25T01:01:57.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-25T01:01:57.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-25T01:01:57.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-03-25T01:01:57.104 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-81310-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-03-25T01:01:59.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-03-25T01:01:59.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-03-25T01:01:59.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-25T01:01:59.610 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-25T01:01:59.611 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-25T01:01:59.612 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-25T01:01:59.613 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-03-25T01:01:59.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-25T01:01:59.614 INFO:tasks.workunit.client.0.smithi080.stderr:start osd.2 2024-03-25T01:01:59.614 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=681b598a-215c-48ce-9d4f-759dddb9c346 --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81310/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-25T01:01:59.615 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-03-25T01:01:59.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-25T01:01:59.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-25T01:01:59.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-25T01:01:59.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-25T01:01:59.654 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:01:59.661+0000 7fb658f047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:01:59.656 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:01:59.665+0000 7fb658f047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:01:59.658 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:01:59.665+0000 7fb658f047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-25T01:02:00.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-25T01:02:00.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-25T01:02:00.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-25T01:02:00.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-25T01:02:00.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-25T01:02:00.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:02:00.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-25T01:02:00.043 INFO:tasks.workunit.client.0.smithi080.stderr:0 2024-03-25T01:02:00.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:02:00.043 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:02:00.212 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:02:00.221+0000 7fb658f047c0 -1 Falling back to public interface 2024-03-25T01:02:00.522 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:02:01.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:02:01.524 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:02:01.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-25T01:02:01.525 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T01:02:01.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:02:01.525 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:02:01.943 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:02:02.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:02:02.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:02:02.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-25T01:02:02.945 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T01:02:02.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:02:02.945 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:02:03.227 INFO:tasks.workunit.client.0.smithi080.stderr:2024-03-25T01:02:03.237+0000 7fb658f047c0 -1 osd.2 49 log_to_monitors true 2024-03-25T01:02:03.349 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-25T01:02:04.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-25T01:02:04.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-25T01:02:04.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-25T01:02:04.351 INFO:tasks.workunit.client.0.smithi080.stderr:3 2024-03-25T01:02:04.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-25T01:02:04.351 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-25T01:02:04.875 INFO:tasks.workunit.client.0.smithi080.stderr:osd.2 up in weight 1 up_from 54 up_thru 40 down_at 51 last_clean_interval [19,49) [v2:127.0.0.1:6818/492504469,v1:127.0.0.1:6819/492504469] [v2:127.0.0.1:6820/492504469,v1:127.0.0.1:6821/492504469] exists,up 7ae1444d-8ac1-4a9a-8795-e9ce74611232 2024-03-25T01:02:04.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-25T01:02:04.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-25T01:02:04.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-25T01:02:04.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-25T01:02:04.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-25T01:02:04.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-25T01:02:04.876 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-25T01:02:04.876 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-25T01:02:04.877 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-25T01:02:04.877 INFO:tasks.workunit.client.0.smithi080.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-25T01:02:04.877 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-25T01:02:04.877 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-25T01:02:04.877 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-25T01:02:04.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-25T01:02:04.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-25T01:02:04.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-25T01:02:04.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-25T01:02:04.986 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-25T01:02:04.987 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-25T01:02:05.553 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-25T01:02:05.553 INFO:tasks.workunit.client.0.smithi080.stderr:1 2024-03-25T01:02:05.554 INFO:tasks.workunit.client.0.smithi080.stderr:2 2024-03-25T01:02:05.554 INFO:tasks.workunit.client.0.smithi080.stderr:3' 2024-03-25T01:02:05.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-25T01:02:05.554 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:02:05.554 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-25T01:02:05.849 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803826 2024-03-25T01:02:05.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803826 2024-03-25T01:02:05.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826' 2024-03-25T01:02:05.850 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:02:05.850 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-25T01:02:06.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561033 2024-03-25T01:02:06.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561033 2024-03-25T01:02:06.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-188978561033' 2024-03-25T01:02:06.010 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:02:06.010 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-25T01:02:06.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233987 2024-03-25T01:02:06.160 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233987 2024-03-25T01:02:06.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-188978561033 2-231928233987' 2024-03-25T01:02:06.161 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-25T01:02:06.161 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-25T01:02:06.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084309 2024-03-25T01:02:06.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084309 2024-03-25T01:02:06.312 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803826 1-188978561033 2-231928233987 3-120259084309' 2024-03-25T01:02:06.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:02:06.313 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803826 2024-03-25T01:02:06.313 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:02:06.313 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-25T01:02:06.314 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803826 2024-03-25T01:02:06.314 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:02:06.315 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803826 2024-03-25T01:02:06.315 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-03-25T01:02:06.315 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.0 seq 25769803826 2024-03-25T01:02:06.316 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-25T01:02:06.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803826 -lt 25769803826 2024-03-25T01:02:06.753 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:02:06.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:02:06.754 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561033 2024-03-25T01:02:06.755 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-25T01:02:06.756 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561033 2024-03-25T01:02:06.756 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:02:06.757 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561033 2024-03-25T01:02:06.758 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561033' 2024-03-25T01:02:06.758 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.1 seq 188978561033 2024-03-25T01:02:06.758 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-25T01:02:07.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561033 -lt 188978561033 2024-03-25T01:02:07.287 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:02:07.288 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-231928233987 2024-03-25T01:02:07.288 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:02:07.289 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-25T01:02:07.290 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-231928233987 2024-03-25T01:02:07.291 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:02:07.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233987 2024-03-25T01:02:07.292 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 231928233987' 2024-03-25T01:02:07.292 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.2 seq 231928233987 2024-03-25T01:02:07.292 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-25T01:02:07.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233987 -lt 231928233987 2024-03-25T01:02:07.716 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-25T01:02:07.717 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084309 2024-03-25T01:02:07.717 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-25T01:02:07.719 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-25T01:02:07.720 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084309 2024-03-25T01:02:07.720 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-25T01:02:07.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084309 2024-03-25T01:02:07.721 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084309' 2024-03-25T01:02:07.721 INFO:tasks.workunit.client.0.smithi080.stderr:waiting osd.3 seq 120259084309 2024-03-25T01:02:07.721 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-25T01:02:08.189 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084309 -lt 120259084309 2024-03-25T01:02:08.189 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-25T01:02:08.190 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T01:02:08.190 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T01:02:08.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-25T01:02:08.741 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-25T01:02:08.742 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-25T01:02:08.742 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-25T01:02:08.742 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-25T01:02:08.742 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-25T01:02:08.742 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-25T01:02:08.743 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-25T01:02:09.142 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-25T01:02:09.142 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-25T01:02:09.143 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-25T01:02:09.143 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-25T01:02:09.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-25T01:02:09.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-25T01:02:09.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-25T01:02:09.709 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-03-25T01:02:09.710 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-03-25T01:02:10.391 INFO:tasks.workunit.client.0.smithi080.stderr:noout is unset 2024-03-25T01:02:10.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81310-1-10 fail 2024-03-25T01:02:10.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-03-25T01:02:10.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-03-25T01:02:10.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81310-1-10 2024-03-25T01:02:10.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-03-25T01:02:10.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-03-25T01:02:10.414 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81310-1-10 td/test-erasure-eio/COPY 2024-03-25T01:02:10.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-03-25T01:02:10.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: return 1 2024-03-25T01:02:10.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: return 1 2024-03-25T01:02:10.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: return 1 2024-03-25T01:02:10.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-03-25T01:02:10.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-eio 1 2024-03-25T01:02:10.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-03-25T01:02:10.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-03-25T01:02:10.615 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-03-25T01:02:10.616 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-25T01:02:10.616 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-25T01:02:10.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-25T01:02:10.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-25T01:02:10.616 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-25T01:02:10.931 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-25T01:02:10.932 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-25T01:02:10.933 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-25T01:02:10.934 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-25T01:02:10.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-25T01:02:10.936 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-25T01:02:10.936 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-25T01:02:10.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T01:02:10.937 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-25T01:02:10.938 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-25T01:02:10.939 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T01:02:10.939 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-25T01:02:10.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-03-25T01:02:10.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-03-25T01:02:10.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-03-25T01:02:10.942 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/test-erasure-eio/mgr.x.log td/test-erasure-eio/mon.a.log td/test-erasure-eio/osd.0.log td/test-erasure-eio/osd.1.log td/test-erasure-eio/osd.2.log td/test-erasure-eio/osd.3.log /home/ubuntu/cephtest/archive/log 2024-03-25T01:02:10.944 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-03-25T01:02:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T01:02:10.970 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-25T01:02:10.970 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-25T01:02:10.971 INFO:tasks.workunit.client.0.smithi080.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81310 2024-03-25T01:02:10.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81310 2024-03-25T01:02:10.971 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-25T01:02:10.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-25T01:02:10.972 INFO:tasks.workunit.client.0.smithi080.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-03-25T01:02:10.973 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2024-03-25T01:02:10.974 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-25T01:02:11.277 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test erasure-code/test-erasure-eio.sh) on smithi080 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9c429fb6abfb5deed62697d56d8997d0f0d6d83f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh' 2024-03-25T01:02:11.522 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1063392cfea941bdb9900a0d513e8955 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_9c429fb6abfb5deed62697d56d8997d0f0d6d83f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test erasure-code/test-erasure-eio.sh) on smithi080 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9c429fb6abfb5deed62697d56d8997d0f0d6d83f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh' 2024-03-25T01:02:11.525 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T01:02:11.535 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T01:02:11.535 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T01:02:11.574 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T01:02:11.582 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T01:02:11.583 DEBUG:teuthology.orchestra.run.smithi080:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T01:02:11.610 INFO:teuthology.orchestra.run.smithi080.stdout: remote refid st t when poll reach delay offset jitter 2024-03-25T01:02:11.610 INFO:teuthology.orchestra.run.smithi080.stdout:============================================================================== 2024-03-25T01:02:11.610 INFO:teuthology.orchestra.run.smithi080.stdout:*hv01.front.sepi 44.4.53.2 2 u 35 64 377 0.097 -1.908 0.232 2024-03-25T01:02:11.610 INFO:teuthology.orchestra.run.smithi080.stdout:+hv02.front.sepi 63.231.80.2 3 u 52 64 377 0.099 -1.697 0.175 2024-03-25T01:02:11.610 INFO:teuthology.orchestra.run.smithi080.stdout:+hv03.front.sepi 74.6.168.72 3 u 33 64 377 0.098 -1.728 0.170 2024-03-25T01:02:11.611 INFO:teuthology.orchestra.run.smithi080.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-25T01:02:11.611 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T01:02:11.655 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T01:02:11.656 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T01:02:11.823 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T01:02:11.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T01:02:11.885 INFO:teuthology.task.internal:Duration was 4468.656921 seconds 2024-03-25T01:02:11.885 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T01:02:11.942 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T01:02:11.943 DEBUG:teuthology.orchestra.run.smithi080:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T01:02:11.987 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T01:02:11.987 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi080.front.sepia.ceph.com 2024-03-25T01:02:11.987 DEBUG:teuthology.orchestra.run.smithi080:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T01:02:12.043 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T01:02:12.044 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T01:02:12.131 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T01:02:12.131 DEBUG:teuthology.orchestra.run.smithi080:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T01:02:12.229 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T01:02:12.238 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T01:02:12.238 DEBUG:teuthology.orchestra.run.smithi080:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T01:02:12.250 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T01:02:12.271 DEBUG:teuthology.orchestra.run.smithi080:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T01:02:12.299 INFO:teuthology.orchestra.run.smithi080.stdout:kernel.core_pattern = core 2024-03-25T01:02:12.328 DEBUG:teuthology.orchestra.run.smithi080:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T01:02:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T01:02:12.375 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T01:02:12.386 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T01:02:12.387 DEBUG:teuthology.misc:Transferring archived files from smithi080:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620593/remote/smithi080 2024-03-25T01:02:12.388 DEBUG:teuthology.orchestra.run.smithi080:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T01:02:13.614 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T01:02:13.614 DEBUG:teuthology.orchestra.run.smithi080:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T01:02:13.663 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T01:02:13.673 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T01:02:13.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T01:02:13.708 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T01:02:13.708 DEBUG:teuthology.orchestra.run.smithi080:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T01:02:13.715 INFO:teuthology.orchestra.run.smithi080.stdout: 658641 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 25 01:02 /home/ubuntu/cephtest 2024-03-25T01:02:13.715 INFO:teuthology.orchestra.run.smithi080.stdout: 671714 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 24 23:56 /home/ubuntu/cephtest/mnt.0 2024-03-25T01:02:13.715 INFO:teuthology.orchestra.run.smithi080.stdout: 673740 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 25 00:17 /home/ubuntu/cephtest/mnt.0/client.0 2024-03-25T01:02:13.715 INFO:teuthology.orchestra.run.smithi080.stdout: 788990 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 25 00:17 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-25T01:02:13.716 INFO:teuthology.orchestra.run.smithi080.stdout: 788991 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 25 01:02 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-03-25T01:02:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T01:02:13.716 INFO:teuthology.orchestra.run.smithi080.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-25T01:02:13.717 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi080 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-25T01:02:13.722 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T01:02:13.754 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T01:02:13.796 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi080 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-25T01:02:13.797 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T01:02:13.824 DEBUG:teuthology.nuke:shortname: smithi080 2024-03-25T01:02:13.825 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T01:02:13.847 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi080.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620593', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 23:39:32.597008', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMr7pn2Kl7j8q7QsNf7SVQKcOEdbGDMAKNatw3P8K0C0uMAJ+Eu1kK8JyWhd/6H4iNZ1wwAJMJN0XMd48o7M2zM='} 2024-03-25T01:02:13.896 INFO:teuthology.orchestra.console.smithi080:Power off 2024-03-25T01:02:13.897 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T01:02:13.919 DEBUG:teuthology.orchestra.console.smithi080:power off output: Chassis Power Control: Down/Off 2024-03-25T01:02:13.919 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T01:02:13.948 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is on 2024-03-25T01:02:17.949 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T01:02:18.061 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is on 2024-03-25T01:02:22.062 DEBUG:teuthology.orchestra.console.smithi080:pexpect command: ipmitool -H smithi080.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T01:02:22.173 DEBUG:teuthology.orchestra.console.smithi080:check power output: Chassis Power is off 2024-03-25T01:02:22.274 INFO:teuthology.orchestra.console.smithi080:Power off completed 2024-03-25T01:02:22.379 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} duration: 4468.6569211483 failure_reason: 'Command failed (workunit test erasure-code/test-erasure-eio.sh) on smithi080 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9c429fb6abfb5deed62697d56d8997d0f0d6d83f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1063392cfea941bdb9900a0d513e8955 status: fail success: false 2024-03-25T01:02:22.381 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T01:02:22.468 INFO:teuthology.run:FAIL